[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:09:12 dist-test-slave-4n6h krb5kdc[6364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:09:12 dist-test-slave-4n6h krb5kdc[6364](info): set up 2 sockets
Jun 20 07:09:12 dist-test-slave-4n6h krb5kdc[6364](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 07:09:13.143926 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.011s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:09:13 dist-test-slave-4n6h krb5kdc[6364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:09:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:09:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:09:13.310077 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:39627
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:43517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:39627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:13.648275 6377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:13.648535 6377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:13.648619 6377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:13.655711 6377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:13.655824 6377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:13.655876 6377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:13.655927 6377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:13.655970 6377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:13.665450 6377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:43517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:39627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:39627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:13.666708 6377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:13.667913 6377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:13.679821 6382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:13.679869 6385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:13.679903 6383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:13.680958 6377 server_base.cc:1034] running on GCE node
I20260620 07:09:13.681734 6377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:13.683233 6377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:13.684468 6377 hybrid_clock.cc:648] HybridClock initialized: now 1781939353684442 us; error 61 us; skew 500 ppm
I20260620 07:09:13.684798 6377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:13 dist-test-slave-4n6h krb5kdc[6364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939353, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:13.687927 6377 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:09:13.689317 6377 webserver.cc:460] Webserver started at http://127.6.53.126:35475/ using document root <none> and password file <none>
I20260620 07:09:13.689940 6377 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:13.690032 6377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:13.690321 6377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:13.693115 6377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "054dc38164c24231a3e250c28a201607"
format_stamp: "Formatted at 2026-06-20 07:09:13 on dist-test-slave-4n6h"
server_key: "f464aa6b0fb1588ab82d44ff39e5990a"
server_key_iv: "1a0c618f1958d7a4b2413fb91ad21d0f"
server_key_version: "encryptionkey@0"
I20260620 07:09:13.693791 6377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "054dc38164c24231a3e250c28a201607"
format_stamp: "Formatted at 2026-06-20 07:09:13 on dist-test-slave-4n6h"
server_key: "f464aa6b0fb1588ab82d44ff39e5990a"
server_key_iv: "1a0c618f1958d7a4b2413fb91ad21d0f"
server_key_version: "encryptionkey@0"
I20260620 07:09:13.698740 6377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:09:13.701844 6392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:13.703172 6377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:13.703361 6377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "054dc38164c24231a3e250c28a201607"
format_stamp: "Formatted at 2026-06-20 07:09:13 on dist-test-slave-4n6h"
server_key: "f464aa6b0fb1588ab82d44ff39e5990a"
server_key_iv: "1a0c618f1958d7a4b2413fb91ad21d0f"
server_key_version: "encryptionkey@0"
I20260620 07:09:13.703547 6377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:13.733839 6377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:13.738233 6377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:13.738582 6377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:13.756711 6377 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:39627
I20260620 07:09:13.756748 6443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:39627 every 8 connection(s)
I20260620 07:09:13.757987 6377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:13.761659 6444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:13.765937 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 6377
I20260620 07:09:13.766263 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:09:13.768540 6357 external_mini_cluster.cc:1428] Setting key de4e8041259b72a092076ed513cfb320
I20260620 07:09:13.773823 6444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607: Bootstrap starting.
I20260620 07:09:13.776716 6444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:13.777626 6444 log.cc:824] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:13.780259 6444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607: No bootstrap required, opened a new log
I20260620 07:09:13.790172 6444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054dc38164c24231a3e250c28a201607" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39627 } }
I20260620 07:09:13.790688 6444 raft_consensus.cc:374] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:13.790800 6444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054dc38164c24231a3e250c28a201607, State: Initialized, Role: FOLLOWER
I20260620 07:09:13.791430 6444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054dc38164c24231a3e250c28a201607" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39627 } }
I20260620 07:09:13.791692 6444 raft_consensus.cc:388] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:09:13.791838 6444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:09:13.792029 6444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:13.794943 6444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054dc38164c24231a3e250c28a201607" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39627 } }
I20260620 07:09:13.795444 6444 leader_election.cc:302] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 054dc38164c24231a3e250c28a201607; no voters:
I20260620 07:09:13.796137 6444 leader_election.cc:288] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:09:13.796262 6449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:13.797607 6449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [term 1 LEADER]: Becoming Leader. State: Replica: 054dc38164c24231a3e250c28a201607, State: Running, Role: LEADER
I20260620 07:09:13.798195 6449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054dc38164c24231a3e250c28a201607" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39627 } }
I20260620 07:09:13.798516 6444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 07:09:13 dist-test-slave-4n6h krb5kdc[6364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:13.804064 6450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "054dc38164c24231a3e250c28a201607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054dc38164c24231a3e250c28a201607" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39627 } } }
I20260620 07:09:13.804428 6450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:13.804883 6451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 054dc38164c24231a3e250c28a201607. Latest consensus state: current_term: 1 leader_uuid: "054dc38164c24231a3e250c28a201607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054dc38164c24231a3e250c28a201607" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39627 } } }
I20260620 07:09:13.805162 6451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:13.807435 6456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:09:13.813743 6456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:09:13.819111 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:13.772921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.6.53.126:39627)
0620 07:09:13.773745 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:13.773760 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:13.775061 (+ 1301us) server_negotiation.cc:407] Connection header received
0620 07:09:13.785313 (+ 10252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:13.785338 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:13.785863 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:13.786372 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:13.788990 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:13.791077 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:13.793125 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:13.794224 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:13.794423 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:13.802497 (+ 8074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:13.802513 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:13.802538 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:13.802602 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:13.808980 (+ 6378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:13.810339 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:13.810351 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:13.810361 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:13.810686 (+ 325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:13.811457 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:13.811465 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:13.811472 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:13.812435 (+ 963us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:13.816432 (+ 3997us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:13.816786 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 07:09:13.817278 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":357,"threads_started":1}
I20260620 07:09:13.824766 6466 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:09:13.826812 6456 catalog_manager.cc:1347] Generated new cluster ID: a88f8ce187324556ae6b87508b17eea8
I20260620 07:09:13.826908 6456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:13.838304 6456 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:13.839986 6456 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:13.849018 6456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 054dc38164c24231a3e250c28a201607: Generated new TSK 0
I20260620 07:09:13.849895 6456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:09:13.897367 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:39627
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:43517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:14.235780 6470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:14.236027 6470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:14.236227 6470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:14.243119 6470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:14.243242 6470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:14.243433 6470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:09:14.252792 6470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:43517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:39627
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:14.253930 6470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:14.255190 6470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:14.267865 6478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:14.268031 6476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:14.267998 6475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:14.268882 6470 server_base.cc:1034] running on GCE node
I20260620 07:09:14.269414 6470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:14.270207 6470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:14.271448 6470 hybrid_clock.cc:648] HybridClock initialized: now 1781939354271402 us; error 40 us; skew 500 ppm
I20260620 07:09:14.271751 6470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:14 dist-test-slave-4n6h krb5kdc[6364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939354, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:14.274863 6470 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:09:14.276259 6470 webserver.cc:460] Webserver started at http://127.6.53.65:41223/ using document root <none> and password file <none>
I20260620 07:09:14.276856 6470 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:14.276957 6470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:14.277208 6470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:14.280135 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "eae0a1d705db47cd966df0e06a5eb11b"
format_stamp: "Formatted at 2026-06-20 07:09:14 on dist-test-slave-4n6h"
server_key: "6502f78ac09faeeed4a49dfcaead2cfd"
server_key_iv: "ede1b129705ce8cd68874890de35f9ea"
server_key_version: "encryptionkey@0"
I20260620 07:09:14.280829 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "eae0a1d705db47cd966df0e06a5eb11b"
format_stamp: "Formatted at 2026-06-20 07:09:14 on dist-test-slave-4n6h"
server_key: "6502f78ac09faeeed4a49dfcaead2cfd"
server_key_iv: "ede1b129705ce8cd68874890de35f9ea"
server_key_version: "encryptionkey@0"
I20260620 07:09:14.285897 6470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:09:14.289587 6485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:14.290982 6470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:14.291143 6470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "eae0a1d705db47cd966df0e06a5eb11b"
format_stamp: "Formatted at 2026-06-20 07:09:14 on dist-test-slave-4n6h"
server_key: "6502f78ac09faeeed4a49dfcaead2cfd"
server_key_iv: "ede1b129705ce8cd68874890de35f9ea"
server_key_version: "encryptionkey@0"
I20260620 07:09:14.291337 6470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:14.308459 6470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:14.312402 6470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:14.312680 6470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:14.313724 6470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:14.315076 6470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:14.315173 6470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:14.315352 6470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:14.315423 6470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:14.348515 6470 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:35399
I20260620 07:09:14.348492 6597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:35399 every 8 connection(s)
I20260620 07:09:14.349735 6470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:09:14.354558 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 6470
I20260620 07:09:14.354790 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:09:14.355545 6357 external_mini_cluster.cc:1428] Setting key 4f28dda0eab584c4fe8eb7d6848706d7
Jun 20 07:09:14 dist-test-slave-4n6h krb5kdc[6364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939354, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
I20260620 07:09:14.373059 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:14.352369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:50075 (local address 127.6.53.126:39627)
0620 07:09:14.352806 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:14.352810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:14.353695 (+ 885us) server_negotiation.cc:407] Connection header received
0620 07:09:14.356368 (+ 2673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:14.356373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:14.356461 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:14.356577 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:14.358230 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:14.358994 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:14.361461 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:14.361894 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:14.361993 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:14.366031 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:14.366039 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:14.366047 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:14.366090 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:14.369836 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:14.370614 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:14.370620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:14.370626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:14.370774 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:14.371186 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:14.371191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:14.371195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:14.371581 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:14.371737 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:14.372395 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 07:09:14.372633 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":247,"threads_started":1}
I20260620 07:09:14.373802 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:14.352865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:39627 (local address 127.6.53.65:50075)
0620 07:09:14.353541 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:14.353577 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:14.356044 (+ 2467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:14.356813 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:14.356821 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:14.357300 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:14.358057 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:14.358082 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:14.359159 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:14.359167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:14.361323 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:14.361341 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:14.362117 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:14.362125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:14.362482 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:14.363195 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:14.363226 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:14.365814 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:14.369992 (+ 4178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:14.369998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:14.370010 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:14.370494 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:14.370894 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:14.370897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:14.370899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:14.371068 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:14.371709 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:14.371715 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:14.372020 (+ 305us) client_negotiation.cc:769] Sending connection context
0620 07:09:14.372233 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 07:09:14.372602 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":269,"threads_started":1}
I20260620 07:09:14.375487 6598 heartbeater.cc:347] Connected to a master server at 127.6.53.126:39627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
I20260620 07:09:14.375778 6598 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:14.376669 6598 heartbeater.cc:511] Master 127.6.53.126:39627 requested a full tablet report, sending...
I20260620 07:09:14.379027 6409 ts_manager.cc:194] Registered new tserver with Master: eae0a1d705db47cd966df0e06a5eb11b (127.6.53.65:35399)
I20260620 07:09:14.380615 6409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:50075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:09:14.397501 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:39627
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:43517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:14.712145 6604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:14.712396 6604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:14.712494 6604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:14.719578 6604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:14.719676 6604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:14.719799 6604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:09:14.729049 6604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:43517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:39627
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:14.730233 6604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:14.731509 6604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:14.744048 6609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:14.744042 6610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:14.744004 6612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:14.744895 6604 server_base.cc:1034] running on GCE node
I20260620 07:09:14.745395 6604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:14.746219 6604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:14.747470 6604 hybrid_clock.cc:648] HybridClock initialized: now 1781939354747408 us; error 65 us; skew 500 ppm
I20260620 07:09:14.747774 6604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:14 dist-test-slave-4n6h krb5kdc[6364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939354, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:14.750756 6604 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:09:14.752111 6604 webserver.cc:460] Webserver started at http://127.6.53.66:46695/ using document root <none> and password file <none>
I20260620 07:09:14.752692 6604 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:14.752794 6604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:14.753034 6604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:14.755739 6604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "81e1de0356534bae84db737fc4860765"
format_stamp: "Formatted at 2026-06-20 07:09:14 on dist-test-slave-4n6h"
server_key: "f8635843c17a927f2568a614193c9f58"
server_key_iv: "cd6b3c510b3f72c6e4833e390e014cc9"
server_key_version: "encryptionkey@0"
I20260620 07:09:14.756567 6604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "81e1de0356534bae84db737fc4860765"
format_stamp: "Formatted at 2026-06-20 07:09:14 on dist-test-slave-4n6h"
server_key: "f8635843c17a927f2568a614193c9f58"
server_key_iv: "cd6b3c510b3f72c6e4833e390e014cc9"
server_key_version: "encryptionkey@0"
I20260620 07:09:14.761446 6604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 07:09:14.765146 6619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:14.766510 6604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:09:14.766662 6604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "81e1de0356534bae84db737fc4860765"
format_stamp: "Formatted at 2026-06-20 07:09:14 on dist-test-slave-4n6h"
server_key: "f8635843c17a927f2568a614193c9f58"
server_key_iv: "cd6b3c510b3f72c6e4833e390e014cc9"
server_key_version: "encryptionkey@0"
I20260620 07:09:14.766851 6604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:14.783821 6604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:14.788043 6604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:14.788323 6604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:14.789240 6604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:14.790578 6604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:14.790678 6604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:14.790830 6604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:14.790903 6604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:14.824046 6604 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:40239
I20260620 07:09:14.824093 6731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:40239 every 8 connection(s)
I20260620 07:09:14.825297 6604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:09:14.832559 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 6604
I20260620 07:09:14.832743 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:09:14.833159 6357 external_mini_cluster.cc:1428] Setting key d2497269eb50b8550f428c3e3316b572
Jun 20 07:09:14 dist-test-slave-4n6h krb5kdc[6364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939354, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:14.846081 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:14.827717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:43419 (local address 127.6.53.126:39627)
0620 07:09:14.827878 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:14.827883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:14.829043 (+ 1160us) server_negotiation.cc:407] Connection header received
0620 07:09:14.830982 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:14.830988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:14.831076 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:14.831188 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:14.832542 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:14.833406 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:14.835271 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:14.835677 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:14.835809 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:14.839458 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:14.839467 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:14.839473 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:14.839514 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:14.843024 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:14.843685 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:14.843693 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:14.843699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:14.843849 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:14.844249 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:14.844256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:14.844262 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:14.844662 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:14.844817 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:14.845446 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 07:09:14.845705 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:09:14.846848 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:14.828178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:39627 (local address 127.6.53.66:43419)
0620 07:09:14.828892 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:14.828933 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:14.830721 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:14.831404 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:14.831412 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:14.831886 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:14.832388 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:14.832404 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:14.833555 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:14.833559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:14.835140 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:14.835151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:14.835897 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:14.835905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:14.836113 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:14.836769 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:14.836793 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:14.839221 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:14.843149 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:14.843155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:14.843167 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:14.843554 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:14.843956 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:14.843959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:14.843961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:14.844127 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:14.844785 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:14.844790 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:14.845068 (+ 278us) client_negotiation.cc:769] Sending connection context
0620 07:09:14.845273 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 07:09:14.845711 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":293,"threads_started":1}
I20260620 07:09:14.848241 6732 heartbeater.cc:347] Connected to a master server at 127.6.53.126:39627
I20260620 07:09:14.848541 6732 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
I20260620 07:09:14.849323 6732 heartbeater.cc:511] Master 127.6.53.126:39627 requested a full tablet report, sending...
I20260620 07:09:14.850776 6409 ts_manager.cc:194] Registered new tserver with Master: 81e1de0356534bae84db737fc4860765 (127.6.53.66:40239)
I20260620 07:09:14.851588 6409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:43419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:09:14.871609 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:39627
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:43517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:15.180936 6737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:15.181234 6737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:15.181412 6737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:15.188488 6737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:15.188621 6737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:15.188802 6737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:09:15.198377 6737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:43517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:39627
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:15.199669 6737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:15.200978 6737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:15.214269 6742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:15.214273 6743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:15.214435 6745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:15.215346 6737 server_base.cc:1034] running on GCE node
I20260620 07:09:15.215919 6737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:15.216779 6737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:15.217967 6737 hybrid_clock.cc:648] HybridClock initialized: now 1781939355217926 us; error 31 us; skew 500 ppm
I20260620 07:09:15.218304 6737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:15 dist-test-slave-4n6h krb5kdc[6364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939355, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:15.221239 6737 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:09:15.222695 6737 webserver.cc:460] Webserver started at http://127.6.53.67:39025/ using document root <none> and password file <none>
I20260620 07:09:15.223335 6737 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:15.223480 6737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:15.223785 6737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:15.226569 6737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "7bff92f0dcdd422f985de243fd52c1ae"
format_stamp: "Formatted at 2026-06-20 07:09:15 on dist-test-slave-4n6h"
server_key: "a42ec70c26b095a938dc1749b44072cb"
server_key_iv: "58771895fc34e1c9bd1453e500cfec27"
server_key_version: "encryptionkey@0"
I20260620 07:09:15.227391 6737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "7bff92f0dcdd422f985de243fd52c1ae"
format_stamp: "Formatted at 2026-06-20 07:09:15 on dist-test-slave-4n6h"
server_key: "a42ec70c26b095a938dc1749b44072cb"
server_key_iv: "58771895fc34e1c9bd1453e500cfec27"
server_key_version: "encryptionkey@0"
I20260620 07:09:15.232264 6737 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 07:09:15.235411 6752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:15.236523 6737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:09:15.236683 6737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "7bff92f0dcdd422f985de243fd52c1ae"
format_stamp: "Formatted at 2026-06-20 07:09:15 on dist-test-slave-4n6h"
server_key: "a42ec70c26b095a938dc1749b44072cb"
server_key_iv: "58771895fc34e1c9bd1453e500cfec27"
server_key_version: "encryptionkey@0"
I20260620 07:09:15.236886 6737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:15.274884 6737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:15.278716 6737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:15.279024 6737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:15.279959 6737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:15.281250 6737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:15.281352 6737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:15.281505 6737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:15.281589 6737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:15.314752 6737 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:38475
I20260620 07:09:15.314837 6864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:38475 every 8 connection(s)
I20260620 07:09:15.316017 6737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:09:15.325911 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 6737
I20260620 07:09:15.326107 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:09:15.326622 6357 external_mini_cluster.cc:1428] Setting key 8e04ed260c9abf8312f63d639e6a58e1
Jun 20 07:09:15 dist-test-slave-4n6h krb5kdc[6364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939355, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:15.335386 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:15.318442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:43291 (local address 127.6.53.126:39627)
0620 07:09:15.318608 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:15.318613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:15.319752 (+ 1139us) server_negotiation.cc:407] Connection header received
0620 07:09:15.321542 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:15.321548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:15.321668 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:15.321783 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:15.323282 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:15.324019 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:15.325490 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:15.326147 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:15.326312 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:15.329943 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:15.329955 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:15.329959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:15.329990 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:15.332505 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:15.333182 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:15.333195 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:15.333197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:15.333315 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:15.333723 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:15.333728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:15.333731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:15.333998 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:15.334149 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:15.334841 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 07:09:15.335058 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:09:15.336231 6867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:15.318936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:39627 (local address 127.6.53.67:43291)
0620 07:09:15.319598 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:15.319633 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:15.321276 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:15.322004 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:15.322011 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:15.322555 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:15.323112 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:15.323129 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:15.324158 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:15.324163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:15.325348 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:15.325359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:15.326398 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:15.326404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:15.326600 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:15.327308 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:15.327330 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:15.329712 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:15.332639 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:15.332649 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:15.332664 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:15.333059 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:15.333427 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:15.333432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:15.333436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:15.333607 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:15.334125 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:15.334132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:15.334500 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 07:09:15.334742 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 07:09:15.335208 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":283,"threads_started":1}
I20260620 07:09:15.337430 6865 heartbeater.cc:347] Connected to a master server at 127.6.53.126:39627
I20260620 07:09:15.337713 6865 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:15.338457 6865 heartbeater.cc:511] Master 127.6.53.126:39627 requested a full tablet report, sending...
I20260620 07:09:15.339912 6409 ts_manager.cc:194] Registered new tserver with Master: 7bff92f0dcdd422f985de243fd52c1ae (127.6.53.67:38475)
I20260620 07:09:15.340823 6409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:43291
I20260620 07:09:15.342931 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 07:09:15 dist-test-slave-4n6h krb5kdc[6364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939355, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:09:15 dist-test-slave-4n6h krb5kdc[6364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939355, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:09:15.384130 6598 heartbeater.cc:503] Master 127.6.53.126:39627 was elected leader, sending a full tablet report...
I20260620 07:09:15.388336 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:15.373158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.6.53.65:35399)
0620 07:09:15.373829 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:15.373837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:15.373867 (+ 30us) server_negotiation.cc:407] Connection header received
0620 07:09:15.373986 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:15.373993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:15.374232 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:15.374437 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:15.375710 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:15.376796 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:15.378040 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:15.378704 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:15.378805 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:15.382558 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:15.382569 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:15.382582 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:15.382615 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:15.385871 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:15.386616 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:15.386622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:15.386624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:15.386733 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:15.387074 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:15.387077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:15.387079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:15.387411 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:15.387627 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:15.387858 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 07:09:15.388036 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":367,"threads_started":1}
W20260620 07:09:15.389524 6513 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50592
Jun 20 07:09:15 dist-test-slave-4n6h krb5kdc[6364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:09:15 dist-test-slave-4n6h krb5kdc[6364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:09:15.431834 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:15.418386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.6.53.65:35399)
0620 07:09:15.418619 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:15.418626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:15.418864 (+ 238us) server_negotiation.cc:407] Connection header received
0620 07:09:15.419077 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:15.419083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:15.419167 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:15.419284 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:15.420537 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:15.421208 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:15.422483 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:15.422939 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:15.423071 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:15.427172 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:15.427177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:15.427180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:15.427217 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:15.429557 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:15.430097 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:15.430101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:15.430103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:15.430213 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:15.430625 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:15.430628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:15.430630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:15.430883 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:15.431058 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:15.431369 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 07:09:15.431570 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:09:15.433341 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 6470
I20260620 07:09:15.445375 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:15.434245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.6.53.65:35399)
0620 07:09:15.434495 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:15.434503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:15.434840 (+ 337us) server_negotiation.cc:407] Connection header received
0620 07:09:15.434950 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:15.434953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:15.435006 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:15.435078 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:15.436154 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:15.436804 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:15.437979 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:15.438434 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:15.438523 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:15.440676 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:15.440680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:15.440683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:15.440709 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:15.443254 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:15.443840 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:15.443843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:15.443845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:15.443961 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:15.444246 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:15.444249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:15.444251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:15.444464 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:15.444626 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:15.444880 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 07:09:15.445016 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:09:15.446079 6593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:15.529382 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 6470
I20260620 07:09:15.545271 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 6604
Jun 20 07:09:15 dist-test-slave-4n6h krb5kdc[6364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:09:15.560672 6879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:15.546110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.6.53.66:40239)
0620 07:09:15.546739 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:15.546746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:15.546773 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:09:15.546841 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:15.546847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:15.547041 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:15.547174 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:15.548313 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:15.549440 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:15.550729 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:15.551298 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:15.551401 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:15.555510 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:15.555521 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:15.555535 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:15.555568 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:15.558333 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:15.558931 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:15.558938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:15.558940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:15.559055 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:15.559406 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:15.559409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:15.559411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:15.559737 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:15.559964 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:15.560202 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 07:09:15.560395 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":380,"threads_started":1}
I20260620 07:09:15.561985 6727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:15.644200 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 6604
I20260620 07:09:15.660280 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 6737
Jun 20 07:09:15 dist-test-slave-4n6h krb5kdc[6364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:09:15.675429 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:15.661190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.6.53.67:38475)
0620 07:09:15.661811 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:15.661817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:15.661845 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:09:15.661902 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:15.661908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:15.662114 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:15.662288 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:15.663489 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:15.664506 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:15.665829 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:15.666424 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:15.666518 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:15.670404 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:15.670415 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:15.670428 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:15.670479 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:15.673195 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:15.673792 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:15.673798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:15.673800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:15.673910 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:15.674213 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:15.674216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:15.674218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:15.674563 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:15.674811 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:15.674993 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 07:09:15.675177 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":305,"threads_started":1}
I20260620 07:09:15.676669 6860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:15.758531 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 6737
I20260620 07:09:15.774108 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 6377
I20260620 07:09:15.775342 6439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:15.878304 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 6377
2026-06-20T07:09:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3831 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:09:15 dist-test-slave-4n6h krb5kdc[6886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:09:15 dist-test-slave-4n6h krb5kdc[6886](info): set up 2 sockets
Jun 20 07:09:15 dist-test-slave-4n6h krb5kdc[6886](info): commencing operation
krb5kdc: starting...
W20260620 07:09:17.010756 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:09:17 dist-test-slave-4n6h krb5kdc[6886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:09:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:09:17.181929 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:38813
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:43181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:38813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:17.493909 6899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:17.494194 6899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:17.494424 6899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:17.501863 6899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:17.501986 6899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:17.502123 6899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:17.502224 6899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:17.502355 6899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:17.512844 6899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:43181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:38813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:38813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:17.514312 6899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:17.515750 6899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:17.528285 6907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:17.528285 6905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:17.528357 6904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:17.529006 6899 server_base.cc:1034] running on GCE node
I20260620 07:09:17.529778 6899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:17.531258 6899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:17.532501 6899 hybrid_clock.cc:648] HybridClock initialized: now 1781939357532468 us; error 51 us; skew 500 ppm
I20260620 07:09:17.532858 6899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:17 dist-test-slave-4n6h krb5kdc[6886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939357, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:17.538558 6899 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:09:17.540066 6899 webserver.cc:460] Webserver started at http://127.6.53.126:40399/ using document root <none> and password file <none>
I20260620 07:09:17.540758 6899 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:17.540896 6899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:17.541210 6899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:17.544126 6899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "b37a7d4893fc43e0a3286e7e95e2f1f9"
format_stamp: "Formatted at 2026-06-20 07:09:17 on dist-test-slave-4n6h"
server_key: "c5779a305ac6386a8abc4082dac22ee6"
server_key_iv: "5ea9893a5e59e1dfc32152abfe4b8843"
server_key_version: "encryptionkey@0"
I20260620 07:09:17.544855 6899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "b37a7d4893fc43e0a3286e7e95e2f1f9"
format_stamp: "Formatted at 2026-06-20 07:09:17 on dist-test-slave-4n6h"
server_key: "c5779a305ac6386a8abc4082dac22ee6"
server_key_iv: "5ea9893a5e59e1dfc32152abfe4b8843"
server_key_version: "encryptionkey@0"
I20260620 07:09:17.549853 6899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 07:09:17.553133 6914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:17.554581 6899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:17.554764 6899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "b37a7d4893fc43e0a3286e7e95e2f1f9"
format_stamp: "Formatted at 2026-06-20 07:09:17 on dist-test-slave-4n6h"
server_key: "c5779a305ac6386a8abc4082dac22ee6"
server_key_iv: "5ea9893a5e59e1dfc32152abfe4b8843"
server_key_version: "encryptionkey@0"
I20260620 07:09:17.554951 6899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:17.570226 6899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:17.574124 6899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:17.574420 6899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:17.592957 6899 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:38813
I20260620 07:09:17.592986 6965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:38813 every 8 connection(s)
I20260620 07:09:17.594343 6899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:17.596935 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 6899
I20260620 07:09:17.597162 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:09:17.597963 6357 external_mini_cluster.cc:1428] Setting key ef5db01a70ec1240a0966aa8f0e804cc
I20260620 07:09:17.598502 6966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:17.612593 6966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9: Bootstrap starting.
Jun 20 07:09:17 dist-test-slave-4n6h krb5kdc[6886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:17.615758 6966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:17.616761 6966 log.cc:824] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:17.619441 6966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9: No bootstrap required, opened a new log
I20260620 07:09:17.621119 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:17.599952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37080 (local address 127.6.53.126:38813)
0620 07:09:17.600689 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:17.600699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:17.600738 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:09:17.602980 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:17.603001 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:17.603391 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:17.603700 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:17.605499 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:17.606859 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:17.608552 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:17.609248 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:17.609378 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:17.613940 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:17.613947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:17.613959 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:17.614000 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:17.617203 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:17.617914 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:17.617921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:17.617927 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:17.618081 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:17.618491 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:17.618497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:17.618500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:17.619030 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:17.619339 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:17.619787 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 07:09:17.620178 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":350,"threads_started":1}
I20260620 07:09:17.627537 6966 raft_consensus.cc:348] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37a7d4893fc43e0a3286e7e95e2f1f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38813 } }
I20260620 07:09:17.627890 6966 raft_consensus.cc:374] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:17.627993 6966 raft_consensus.cc:729] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37a7d4893fc43e0a3286e7e95e2f1f9, State: Initialized, Role: FOLLOWER
I20260620 07:09:17.628504 6966 consensus_queue.cc:260] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37a7d4893fc43e0a3286e7e95e2f1f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38813 } }
I20260620 07:09:17.628746 6966 raft_consensus.cc:388] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:09:17.628885 6966 raft_consensus.cc:482] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:09:17.629050 6966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:17.631383 6966 raft_consensus.cc:504] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37a7d4893fc43e0a3286e7e95e2f1f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38813 } }
I20260620 07:09:17.631839 6966 leader_election.cc:302] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b37a7d4893fc43e0a3286e7e95e2f1f9; no voters:
I20260620 07:09:17.632369 6966 leader_election.cc:288] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:09:17.632491 6971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:17.633206 6971 raft_consensus.cc:686] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [term 1 LEADER]: Becoming Leader. State: Replica: b37a7d4893fc43e0a3286e7e95e2f1f9, State: Running, Role: LEADER
I20260620 07:09:17.633730 6971 consensus_queue.cc:237] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37a7d4893fc43e0a3286e7e95e2f1f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38813 } }
I20260620 07:09:17.634413 6966 sys_catalog.cc:564] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:17.636471 6972 sys_catalog.cc:455] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b37a7d4893fc43e0a3286e7e95e2f1f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37a7d4893fc43e0a3286e7e95e2f1f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38813 } } }
I20260620 07:09:17.636780 6972 sys_catalog.cc:458] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:17.638636 6976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:09:17.638473 6973 sys_catalog.cc:455] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b37a7d4893fc43e0a3286e7e95e2f1f9. Latest consensus state: current_term: 1 leader_uuid: "b37a7d4893fc43e0a3286e7e95e2f1f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b37a7d4893fc43e0a3286e7e95e2f1f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38813 } } }
I20260620 07:09:17.638794 6973 sys_catalog.cc:458] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:17.643664 6976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:09:17.653056 6976 catalog_manager.cc:1347] Generated new cluster ID: 40dfb6e2d5cf49629f28ca83a460fe0c
I20260620 07:09:17.653162 6976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:17.685720 6976 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:17.687122 6976 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:17.693542 6976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b37a7d4893fc43e0a3286e7e95e2f1f9: Generated new TSK 0
I20260620 07:09:17.694370 6976 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:09:17.747216 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38813
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:43181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:18.071135 6992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:18.071420 6992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:18.071521 6992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:18.078701 6992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:18.078819 6992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:18.078948 6992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:09:18.088522 6992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:43181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38813
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:18.089691 6992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:18.091027 6992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:18.104192 6997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:18.104286 6998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:18.104308 7000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:18.104826 6992 server_base.cc:1034] running on GCE node
I20260620 07:09:18.105336 6992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:18.106164 6992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:18.107363 6992 hybrid_clock.cc:648] HybridClock initialized: now 1781939358107330 us; error 35 us; skew 500 ppm
I20260620 07:09:18.107625 6992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:18 dist-test-slave-4n6h krb5kdc[6886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939358, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:18.110692 6992 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:09:18.112046 6992 webserver.cc:460] Webserver started at http://127.6.53.65:44279/ using document root <none> and password file <none>
I20260620 07:09:18.112715 6992 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:18.112846 6992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:18.113157 6992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:18.116075 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "f9ef73f81349480f9151fc0b7c37cb48"
format_stamp: "Formatted at 2026-06-20 07:09:18 on dist-test-slave-4n6h"
server_key: "532faeff799116e1a75b71fa10d8052e"
server_key_iv: "51787f4dfaa69128e5438188902d94a2"
server_key_version: "encryptionkey@0"
I20260620 07:09:18.116883 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "f9ef73f81349480f9151fc0b7c37cb48"
format_stamp: "Formatted at 2026-06-20 07:09:18 on dist-test-slave-4n6h"
server_key: "532faeff799116e1a75b71fa10d8052e"
server_key_iv: "51787f4dfaa69128e5438188902d94a2"
server_key_version: "encryptionkey@0"
I20260620 07:09:18.122123 6992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 07:09:18.125376 7007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:18.126736 6992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:18.126896 6992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "f9ef73f81349480f9151fc0b7c37cb48"
format_stamp: "Formatted at 2026-06-20 07:09:18 on dist-test-slave-4n6h"
server_key: "532faeff799116e1a75b71fa10d8052e"
server_key_iv: "51787f4dfaa69128e5438188902d94a2"
server_key_version: "encryptionkey@0"
I20260620 07:09:18.127082 6992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:18.145814 6992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:18.149537 6992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:18.149813 6992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:18.150839 6992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:18.152119 6992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:18.152236 6992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:18.152380 6992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:18.152452 6992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:18.186465 6992 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:40079
I20260620 07:09:18.186496 7119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:40079 every 8 connection(s)
I20260620 07:09:18.187777 6992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:09:18.193240 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 6992
I20260620 07:09:18.193476 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:09:18.194321 6357 external_mini_cluster.cc:1428] Setting key 790584d553bb3ccb8d715bd03af22f04
Jun 20 07:09:18 dist-test-slave-4n6h krb5kdc[6886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939358, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:18.207696 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:18.190296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:33373 (local address 127.6.53.126:38813)
0620 07:09:18.190708 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:18.190713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:18.191702 (+ 989us) server_negotiation.cc:407] Connection header received
0620 07:09:18.193690 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:18.193697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:18.193836 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:18.193958 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:18.195714 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:18.196739 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:18.198306 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:18.198698 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:18.198834 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:18.202146 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:18.202151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:18.202154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:18.202184 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:18.204748 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:18.205433 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:18.205440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:18.205449 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:18.205558 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:18.205983 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:18.205988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:18.205991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:18.206304 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:18.206452 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:18.207119 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 07:09:18.207341 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":208,"threads_started":1}
I20260620 07:09:18.208585 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:18.190866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:38813 (local address 127.6.53.65:33373)
0620 07:09:18.191539 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:18.191584 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:18.193410 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:18.194150 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:18.194157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:18.194878 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:18.195531 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:18.195550 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:18.196881 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:18.196887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:18.198140 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:18.198151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:18.198919 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:18.198927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:18.199150 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:18.199849 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:18.199873 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:18.201923 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:18.204882 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:18.204891 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:18.204907 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:18.205308 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:18.205672 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:18.205677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:18.205681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:18.205867 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:18.206424 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:18.206432 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:18.206759 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 07:09:18.207009 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 07:09:18.207515 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":255,"threads_started":1}
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
I20260620 07:09:18.210004 7120 heartbeater.cc:347] Connected to a master server at 127.6.53.126:38813
I20260620 07:09:18.210390 7120 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:18.211136 7120 heartbeater.cc:511] Master 127.6.53.126:38813 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
I20260620 07:09:18.213317 6931 ts_manager.cc:194] Registered new tserver with Master: f9ef73f81349480f9151fc0b7c37cb48 (127.6.53.65:40079)
I20260620 07:09:18.214792 6931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:33373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:09:18.233170 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38813
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:43181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:18.543155 7126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:18.543424 7126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:18.543582 7126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:18.550755 7126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:18.550863 7126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:18.551059 7126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:09:18.560822 7126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:43181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38813
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:18.562161 7126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:18.563524 7126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:18.576296 7132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:18.576244 7134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:18.576308 7131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:18.576939 7126 server_base.cc:1034] running on GCE node
I20260620 07:09:18.577503 7126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:18.578472 7126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:18.579720 7126 hybrid_clock.cc:648] HybridClock initialized: now 1781939358579650 us; error 57 us; skew 500 ppm
I20260620 07:09:18.580039 7126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:18 dist-test-slave-4n6h krb5kdc[6886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939358, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:18.583455 7126 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:09:18.584966 7126 webserver.cc:460] Webserver started at http://127.6.53.66:34595/ using document root <none> and password file <none>
I20260620 07:09:18.585623 7126 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:18.585765 7126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:18.586062 7126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:18.589133 7126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "70d7d7abb282404081eb7dc7a644882d"
format_stamp: "Formatted at 2026-06-20 07:09:18 on dist-test-slave-4n6h"
server_key: "cd32aa45499fed59f12cf80a4b9a2fb3"
server_key_iv: "9342fd8b0aedd56948528a199ee6f459"
server_key_version: "encryptionkey@0"
I20260620 07:09:18.589887 7126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "70d7d7abb282404081eb7dc7a644882d"
format_stamp: "Formatted at 2026-06-20 07:09:18 on dist-test-slave-4n6h"
server_key: "cd32aa45499fed59f12cf80a4b9a2fb3"
server_key_iv: "9342fd8b0aedd56948528a199ee6f459"
server_key_version: "encryptionkey@0"
I20260620 07:09:18.594926 7126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:09:18.598270 7141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:18.599540 7126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:09:18.599702 7126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "70d7d7abb282404081eb7dc7a644882d"
format_stamp: "Formatted at 2026-06-20 07:09:18 on dist-test-slave-4n6h"
server_key: "cd32aa45499fed59f12cf80a4b9a2fb3"
server_key_iv: "9342fd8b0aedd56948528a199ee6f459"
server_key_version: "encryptionkey@0"
I20260620 07:09:18.599896 7126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:18.624250 7126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:18.628268 7126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:18.628551 7126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:18.629444 7126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:18.631187 7126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:18.631294 7126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:18.631438 7126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:18.631507 7126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:18.665084 7126 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:40599
I20260620 07:09:18.665133 7253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:40599 every 8 connection(s)
I20260620 07:09:18.666386 7126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:09:18.668747 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 7126
I20260620 07:09:18.668938 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:09:18.669469 6357 external_mini_cluster.cc:1428] Setting key e718806f63b5c773db06d22061b00599
Jun 20 07:09:18 dist-test-slave-4n6h krb5kdc[6886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939358, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
I20260620 07:09:18.686674 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:18.668940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:32775 (local address 127.6.53.126:38813)
0620 07:09:18.669114 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:18.669118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:18.670749 (+ 1631us) server_negotiation.cc:407] Connection header received
0620 07:09:18.672840 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:18.672845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:18.672938 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:18.673048 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:18.674568 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:18.675268 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:18.676844 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:18.677253 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:18.677383 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:18.680932 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:18.680938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:18.680941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:18.680970 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:18.683535 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:18.684262 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:18.684272 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:18.684274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:18.684390 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:18.684802 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:18.684807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:18.684811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:18.685064 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:18.685216 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:18.686070 (+ 854us) server_negotiation.cc:299] Negotiation successful
0620 07:09:18.686331 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
I20260620 07:09:18.687878 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:18.669620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:38813 (local address 127.6.53.66:32775)
0620 07:09:18.670572 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:18.670635 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:18.672585 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:18.673287 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:18.673295 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:18.673780 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:18.674406 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:18.674424 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:18.675427 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:18.675431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:18.676698 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:18.676709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:18.677462 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:18.677469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:18.677830 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:18.678691 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:18.678712 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:18.680722 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:18.683668 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:18.683720 (+ 52us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:18.683737 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:18.684130 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:18.684496 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:18.684502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:18.684505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:18.684683 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:18.685199 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:18.685207 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:18.685521 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 07:09:18.685979 (+ 458us) client_negotiation.cc:240] Negotiation successful
0620 07:09:18.686529 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":366,"threads_started":1}
I20260620 07:09:18.689293 7254 heartbeater.cc:347] Connected to a master server at 127.6.53.126:38813
I20260620 07:09:18.689644 7254 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:18.690606 7254 heartbeater.cc:511] Master 127.6.53.126:38813 requested a full tablet report, sending...
I20260620 07:09:18.692400 6931 ts_manager.cc:194] Registered new tserver with Master: 70d7d7abb282404081eb7dc7a644882d (127.6.53.66:40599)
I20260620 07:09:18.693408 6931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:32775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:09:18.710067 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38813
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:43181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:19.032837 7259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:19.033103 7259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:19.033198 7259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:19.040472 7259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:19.040572 7259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:19.040691 7259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:09:19.050213 7259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:43181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38813
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:19.051450 7259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:19.052659 7259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:19.065395 7264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:19.065431 7267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:19.065351 7265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:19.066241 7259 server_base.cc:1034] running on GCE node
I20260620 07:09:19.066879 7259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:19.067751 7259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:19.068974 7259 hybrid_clock.cc:648] HybridClock initialized: now 1781939359068931 us; error 54 us; skew 500 ppm
I20260620 07:09:19.069288 7259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:19 dist-test-slave-4n6h krb5kdc[6886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939359, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:19.072373 7259 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:09:19.073875 7259 webserver.cc:460] Webserver started at http://127.6.53.67:41277/ using document root <none> and password file <none>
I20260620 07:09:19.074596 7259 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:19.074741 7259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:19.075034 7259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:19.077766 7259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "7b46d70349f041eb928c6a148933aa42"
format_stamp: "Formatted at 2026-06-20 07:09:19 on dist-test-slave-4n6h"
server_key: "cf552bdd4798222c0812b10caf2f4d79"
server_key_iv: "f218c15a49c1fbe082552ad72ce14154"
server_key_version: "encryptionkey@0"
I20260620 07:09:19.078583 7259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "7b46d70349f041eb928c6a148933aa42"
format_stamp: "Formatted at 2026-06-20 07:09:19 on dist-test-slave-4n6h"
server_key: "cf552bdd4798222c0812b10caf2f4d79"
server_key_iv: "f218c15a49c1fbe082552ad72ce14154"
server_key_version: "encryptionkey@0"
I20260620 07:09:19.083447 7259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:09:19.086926 7274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:19.088271 7259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:09:19.088464 7259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "7b46d70349f041eb928c6a148933aa42"
format_stamp: "Formatted at 2026-06-20 07:09:19 on dist-test-slave-4n6h"
server_key: "cf552bdd4798222c0812b10caf2f4d79"
server_key_iv: "f218c15a49c1fbe082552ad72ce14154"
server_key_version: "encryptionkey@0"
I20260620 07:09:19.088696 7259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:19.105425 7259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:19.109822 7259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:19.110122 7259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:19.111163 7259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:19.112464 7259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:19.112576 7259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:19.112763 7259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:19.112844 7259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:19.147178 7259 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:35165
I20260620 07:09:19.147217 7386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:35165 every 8 connection(s)
I20260620 07:09:19.148486 7259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:09:19.151253 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 7259
I20260620 07:09:19.151438 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:09:19.151998 6357 external_mini_cluster.cc:1428] Setting key e57f01f76db2080622389b2685056753
Jun 20 07:09:19 dist-test-slave-4n6h krb5kdc[6886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939359, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:19.168598 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.150969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:37949 (local address 127.6.53.126:38813)
0620 07:09:19.151136 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:19.151140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:19.152577 (+ 1437us) server_negotiation.cc:407] Connection header received
0620 07:09:19.154936 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:19.154942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:19.155047 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:19.155131 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:19.156546 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.157254 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.158848 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.159353 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.159494 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.162613 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:19.162617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:19.162621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:19.162650 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:19.165289 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:19.166245 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:19.166269 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:19.166273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:19.166413 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:19.166946 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:19.166950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:19.166953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:19.167335 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:19.167482 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:19.168208 (+ 726us) server_negotiation.cc:299] Negotiation successful
0620 07:09:19.168363 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:09:19.169552 7389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.151618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:38813 (local address 127.6.53.67:37949)
0620 07:09:19.152430 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:19.152466 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:19.154640 (+ 2174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:19.155371 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:19.155378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:19.155857 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:19.156402 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:19.156419 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.157381 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.157385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:19.158729 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:19.158742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.159549 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.159554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:19.159736 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.160338 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:19.160358 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:19.162440 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:19.165435 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:19.165444 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:19.165458 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:19.166097 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:19.166529 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:19.166534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:19.166538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:19.166796 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:19.167496 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:19.167504 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:19.167875 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 07:09:19.168101 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 07:09:19.168460 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":349,"threads_started":1}
I20260620 07:09:19.171007 7387 heartbeater.cc:347] Connected to a master server at 127.6.53.126:38813
I20260620 07:09:19.171339 7387 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:19.172083 7387 heartbeater.cc:511] Master 127.6.53.126:38813 requested a full tablet report, sending...
I20260620 07:09:19.173457 6931 ts_manager.cc:194] Registered new tserver with Master: 7b46d70349f041eb928c6a148933aa42 (127.6.53.67:35165)
I20260620 07:09:19.174345 6931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:37949
I20260620 07:09:19.179673 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:09:19.199069 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.185392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37092 (local address 127.6.53.126:38813)
0620 07:09:19.185657 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:19.185665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:19.186285 (+ 620us) server_negotiation.cc:407] Connection header received
0620 07:09:19.186418 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:19.186421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:19.186476 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:19.186555 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:19.188428 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.189177 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.191051 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.191473 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.191609 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.194020 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:19.194025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:19.194028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:19.194060 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:19.196665 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:19.197232 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:19.197236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:19.197238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:19.197363 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:19.197699 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:19.197702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:19.197703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:19.198002 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:19.198188 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:19.198505 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 07:09:19.198702 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:09:19.205277 6931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:09:19.211831 6931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:09:19.217511 7120 heartbeater.cc:503] Master 127.6.53.126:38813 was elected leader, sending a full tablet report...
I20260620 07:09:19.235159 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.223712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.6.53.65:40079)
0620 07:09:19.224428 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:19.224440 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:19.225289 (+ 849us) server_negotiation.cc:407] Connection header received
0620 07:09:19.226040 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:19.226048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:19.226324 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:19.226486 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:19.227299 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.228903 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.232281 (+ 3378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.233695 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.233786 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.234007 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:19.234657 (+ 650us) server_negotiation.cc:299] Negotiation successful
0620 07:09:19.234824 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":746,"threads_started":1}
I20260620 07:09:19.235929 7399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.223504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:40079 (local address 127.0.0.1:58256)
0620 07:09:19.225156 (+ 1652us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:19.225197 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:19.225438 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:19.226662 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:19.226668 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:19.226695 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:19.227087 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:19.227100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.229508 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.229513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:19.232097 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:19.232111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.234176 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.234182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:19.234401 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.234457 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 07:09:19.235359 (+ 902us) client_negotiation.cc:240] Negotiation successful
0620 07:09:19.235540 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1428,"thread_start_us":1294,"threads_started":1}
I20260620 07:09:19.237572 7401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.223819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:40599 (local address 127.0.0.1:46332)
0620 07:09:19.226328 (+ 2509us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:19.226347 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:19.226461 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:19.227921 (+ 1460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:19.227926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:19.227949 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:19.228354 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:19.228367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.230927 (+ 2560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.230939 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:19.234018 (+ 3079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:19.234034 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.236727 (+ 2693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.236750 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:19.236939 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.236969 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:09:19.237056 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 07:09:19.237176 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2024,"mutex_wait_us":1,"thread_start_us":3241,"threads_started":1}
I20260620 07:09:19.238281 7402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.224348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46332 (local address 127.6.53.66:40599)
0620 07:09:19.227144 (+ 2796us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:19.227153 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:19.227179 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:09:19.227270 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:19.227278 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:19.227502 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:19.227678 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:19.228556 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.230659 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.234185 (+ 3526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.235959 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.236089 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.236288 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:19.237790 (+ 1502us) server_negotiation.cc:299] Negotiation successful
0620 07:09:19.237954 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2635,"thread_start_us":2489,"threads_started":1}
I20260620 07:09:19.240119 7055 tablet_service.cc:1467] Processing CreateTablet for tablet 3f405483ce8b4cfea16c161596b64152 (DEFAULT_TABLE table=test-table [id=2a6a7b8e70e445a0bc83b5afaf1188e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:19.241765 7055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f405483ce8b4cfea16c161596b64152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:19.241444 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 3f405483ce8b4cfea16c161596b64152 (DEFAULT_TABLE table=test-table [id=2a6a7b8e70e445a0bc83b5afaf1188e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:19.242882 7189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f405483ce8b4cfea16c161596b64152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:19.246811 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.226146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:35165 (local address 127.0.0.1:56856)
0620 07:09:19.236798 (+ 10652us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:19.236829 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:19.236989 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:19.239297 (+ 2308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:19.239302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:19.239325 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:19.239637 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:19.239648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.241758 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.241764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:19.244096 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:19.244115 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.246024 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.246030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:19.246198 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.246232 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 07:09:19.246352 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 07:09:19.246476 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10457,"thread_start_us":10962,"threads_started":1}
I20260620 07:09:19.246961 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.226903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.6.53.67:35165)
0620 07:09:19.238000 (+ 11097us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:19.238006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:19.238035 (+ 29us) server_negotiation.cc:407] Connection header received
0620 07:09:19.238113 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:19.238118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:19.238361 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:19.239839 (+ 1478us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:19.240102 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.241603 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.244318 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.245816 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.245961 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.246189 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:19.246398 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 07:09:19.246560 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10960,"thread_start_us":10656,"threads_started":1}
I20260620 07:09:19.250613 7322 tablet_service.cc:1467] Processing CreateTablet for tablet 3f405483ce8b4cfea16c161596b64152 (DEFAULT_TABLE table=test-table [id=2a6a7b8e70e445a0bc83b5afaf1188e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:19.251950 7322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f405483ce8b4cfea16c161596b64152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:19.254796 7405 tablet_bootstrap.cc:492] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48: Bootstrap starting.
I20260620 07:09:19.256477 7406 tablet_bootstrap.cc:492] T 3f405483ce8b4cfea16c161596b64152 P 70d7d7abb282404081eb7dc7a644882d: Bootstrap starting.
I20260620 07:09:19.258976 7405 tablet_bootstrap.cc:654] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:19.260273 7405 log.cc:824] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:19.260465 7406 tablet_bootstrap.cc:654] T 3f405483ce8b4cfea16c161596b64152 P 70d7d7abb282404081eb7dc7a644882d: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:19.261628 7406 log.cc:824] T 3f405483ce8b4cfea16c161596b64152 P 70d7d7abb282404081eb7dc7a644882d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:19.263327 7405 tablet_bootstrap.cc:492] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48: No bootstrap required, opened a new log
I20260620 07:09:19.263593 7405 ts_tablet_manager.cc:1397] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 07:09:19.264571 7406 tablet_bootstrap.cc:492] T 3f405483ce8b4cfea16c161596b64152 P 70d7d7abb282404081eb7dc7a644882d: No bootstrap required, opened a new log
I20260620 07:09:19.264846 7406 ts_tablet_manager.cc:1397] T 3f405483ce8b4cfea16c161596b64152 P 70d7d7abb282404081eb7dc7a644882d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 07:09:19.266952 7409 tablet_bootstrap.cc:492] T 3f405483ce8b4cfea16c161596b64152 P 7b46d70349f041eb928c6a148933aa42: Bootstrap starting.
I20260620 07:09:19.270537 7409 tablet_bootstrap.cc:654] T 3f405483ce8b4cfea16c161596b64152 P 7b46d70349f041eb928c6a148933aa42: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:19.271752 7409 log.cc:824] T 3f405483ce8b4cfea16c161596b64152 P 7b46d70349f041eb928c6a148933aa42: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:19.272677 7406 raft_consensus.cc:348] T 3f405483ce8b4cfea16c161596b64152 P 70d7d7abb282404081eb7dc7a644882d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ef73f81349480f9151fc0b7c37cb48" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 40079 } } peers { permanent_uuid: "70d7d7abb282404081eb7dc7a644882d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 40599 } } peers { permanent_uuid: "7b46d70349f041eb928c6a148933aa42" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35165 } }
I20260620 07:09:19.273319 7406 raft_consensus.cc:374] T 3f405483ce8b4cfea16c161596b64152 P 70d7d7abb282404081eb7dc7a644882d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:19.273434 7406 raft_consensus.cc:729] T 3f405483ce8b4cfea16c161596b64152 P 70d7d7abb282404081eb7dc7a644882d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d7d7abb282404081eb7dc7a644882d, State: Initialized, Role: FOLLOWER
I20260620 07:09:19.274041 7406 consensus_queue.cc:260] T 3f405483ce8b4cfea16c161596b64152 P 70d7d7abb282404081eb7dc7a644882d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ef73f81349480f9151fc0b7c37cb48" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 40079 } } peers { permanent_uuid: "70d7d7abb282404081eb7dc7a644882d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 40599 } } peers { permanent_uuid: "7b46d70349f041eb928c6a148933aa42" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35165 } }
I20260620 07:09:19.274214 7405 raft_consensus.cc:348] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ef73f81349480f9151fc0b7c37cb48" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 40079 } } peers { permanent_uuid: "70d7d7abb282404081eb7dc7a644882d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 40599 } } peers { permanent_uuid: "7b46d70349f041eb928c6a148933aa42" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35165 } }
I20260620 07:09:19.274775 7405 raft_consensus.cc:374] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:19.274875 7409 tablet_bootstrap.cc:492] T 3f405483ce8b4cfea16c161596b64152 P 7b46d70349f041eb928c6a148933aa42: No bootstrap required, opened a new log
I20260620 07:09:19.275187 7409 ts_tablet_manager.cc:1397] T 3f405483ce8b4cfea16c161596b64152 P 7b46d70349f041eb928c6a148933aa42: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260620 07:09:19.275151 7254 heartbeater.cc:503] Master 127.6.53.126:38813 was elected leader, sending a full tablet report...
I20260620 07:09:19.275710 7406 ts_tablet_manager.cc:1428] T 3f405483ce8b4cfea16c161596b64152 P 70d7d7abb282404081eb7dc7a644882d: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260620 07:09:19.274928 7405 raft_consensus.cc:729] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9ef73f81349480f9151fc0b7c37cb48, State: Initialized, Role: FOLLOWER
I20260620 07:09:19.276494 7405 consensus_queue.cc:260] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ef73f81349480f9151fc0b7c37cb48" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 40079 } } peers { permanent_uuid: "70d7d7abb282404081eb7dc7a644882d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 40599 } } peers { permanent_uuid: "7b46d70349f041eb928c6a148933aa42" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35165 } }
I20260620 07:09:19.278185 7405 ts_tablet_manager.cc:1428] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260620 07:09:19.284441 7412 raft_consensus.cc:482] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:09:19.284673 7412 raft_consensus.cc:504] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ef73f81349480f9151fc0b7c37cb48" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 40079 } } peers { permanent_uuid: "70d7d7abb282404081eb7dc7a644882d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 40599 } } peers { permanent_uuid: "7b46d70349f041eb928c6a148933aa42" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35165 } }
I20260620 07:09:19.285384 7409 raft_consensus.cc:348] T 3f405483ce8b4cfea16c161596b64152 P 7b46d70349f041eb928c6a148933aa42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ef73f81349480f9151fc0b7c37cb48" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 40079 } } peers { permanent_uuid: "70d7d7abb282404081eb7dc7a644882d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 40599 } } peers { permanent_uuid: "7b46d70349f041eb928c6a148933aa42" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35165 } }
I20260620 07:09:19.285787 7409 raft_consensus.cc:374] T 3f405483ce8b4cfea16c161596b64152 P 7b46d70349f041eb928c6a148933aa42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:19.285914 7409 raft_consensus.cc:729] T 3f405483ce8b4cfea16c161596b64152 P 7b46d70349f041eb928c6a148933aa42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b46d70349f041eb928c6a148933aa42, State: Initialized, Role: FOLLOWER
I20260620 07:09:19.286501 7412 leader_election.cc:288] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70d7d7abb282404081eb7dc7a644882d (127.6.53.66:40599), 7b46d70349f041eb928c6a148933aa42 (127.6.53.67:35165)
I20260620 07:09:19.286592 7409 consensus_queue.cc:260] T 3f405483ce8b4cfea16c161596b64152 P 7b46d70349f041eb928c6a148933aa42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ef73f81349480f9151fc0b7c37cb48" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 40079 } } peers { permanent_uuid: "70d7d7abb282404081eb7dc7a644882d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 40599 } } peers { permanent_uuid: "7b46d70349f041eb928c6a148933aa42" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35165 } }
I20260620 07:09:19.291621 7387 heartbeater.cc:503] Master 127.6.53.126:38813 was elected leader, sending a full tablet report...
I20260620 07:09:19.292624 7409 ts_tablet_manager.cc:1428] T 3f405483ce8b4cfea16c161596b64152 P 7b46d70349f041eb928c6a148933aa42: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260620 07:09:19.293920 7402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.287068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:49647 (local address 127.6.53.66:40599)
0620 07:09:19.287244 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:19.287251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:19.287318 (+ 67us) server_negotiation.cc:407] Connection header received
0620 07:09:19.287544 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:19.287549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:19.287619 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:19.287714 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:19.288218 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.289158 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.291730 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.292894 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.293015 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.293096 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:19.293543 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 07:09:19.293664 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:09:19.294003 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.286580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:40599 (local address 127.6.53.65:49647)
0620 07:09:19.287220 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:19.287249 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:19.287422 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:19.287731 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:19.287736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:19.287761 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:19.288089 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:19.288097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.289289 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.289295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:19.291538 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:19.291553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.293251 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.293255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:19.293373 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.293397 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:09:19.293519 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 07:09:19.293642 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":736,"threads_started":1}
I20260620 07:09:19.295120 7209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f405483ce8b4cfea16c161596b64152" candidate_uuid: "f9ef73f81349480f9151fc0b7c37cb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d7d7abb282404081eb7dc7a644882d" is_pre_election: true
I20260620 07:09:19.295646 7209 raft_consensus.cc:2454] T 3f405483ce8b4cfea16c161596b64152 P 70d7d7abb282404081eb7dc7a644882d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9ef73f81349480f9151fc0b7c37cb48 in term 0.
I20260620 07:09:19.296346 7011 leader_election.cc:302] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70d7d7abb282404081eb7dc7a644882d, f9ef73f81349480f9151fc0b7c37cb48; no voters:
I20260620 07:09:19.296712 7412 raft_consensus.cc:2781] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:09:19.296837 7412 raft_consensus.cc:482] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:09:19.296923 7412 raft_consensus.cc:3037] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:19.297960 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.286964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:35165 (local address 127.6.53.65:59331)
0620 07:09:19.290730 (+ 3766us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:19.290755 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:19.290870 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:19.291441 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:19.291453 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:19.291490 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:19.292035 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:19.292051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.293768 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.293774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:19.295703 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:19.295716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.297387 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.297392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:19.297510 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.297535 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 07:09:19.297615 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 07:09:19.297715 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3634,"thread_start_us":3479,"threads_started":1}
I20260620 07:09:19.298031 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.287697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:59331 (local address 127.6.53.67:35165)
0620 07:09:19.287956 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:19.287966 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:19.290852 (+ 2886us) server_negotiation.cc:407] Connection header received
0620 07:09:19.290999 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:19.291008 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:19.291154 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:19.291271 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:19.292252 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.293375 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.295901 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.297145 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.297304 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.297407 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:19.297659 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 07:09:19.297801 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 07:09:19.298898 7342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f405483ce8b4cfea16c161596b64152" candidate_uuid: "f9ef73f81349480f9151fc0b7c37cb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b46d70349f041eb928c6a148933aa42" is_pre_election: true
I20260620 07:09:19.299343 7342 raft_consensus.cc:2454] T 3f405483ce8b4cfea16c161596b64152 P 7b46d70349f041eb928c6a148933aa42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9ef73f81349480f9151fc0b7c37cb48 in term 0.
I20260620 07:09:19.299708 7412 raft_consensus.cc:504] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ef73f81349480f9151fc0b7c37cb48" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 40079 } } peers { permanent_uuid: "70d7d7abb282404081eb7dc7a644882d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 40599 } } peers { permanent_uuid: "7b46d70349f041eb928c6a148933aa42" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35165 } }
I20260620 07:09:19.300285 7412 leader_election.cc:288] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [CANDIDATE]: Term 1 election: Requested vote from peers 70d7d7abb282404081eb7dc7a644882d (127.6.53.66:40599), 7b46d70349f041eb928c6a148933aa42 (127.6.53.67:35165)
I20260620 07:09:19.300671 7342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f405483ce8b4cfea16c161596b64152" candidate_uuid: "f9ef73f81349480f9151fc0b7c37cb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b46d70349f041eb928c6a148933aa42"
I20260620 07:09:19.300671 7209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f405483ce8b4cfea16c161596b64152" candidate_uuid: "f9ef73f81349480f9151fc0b7c37cb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d7d7abb282404081eb7dc7a644882d"
I20260620 07:09:19.300856 7342 raft_consensus.cc:3037] T 3f405483ce8b4cfea16c161596b64152 P 7b46d70349f041eb928c6a148933aa42 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:19.300863 7209 raft_consensus.cc:3037] T 3f405483ce8b4cfea16c161596b64152 P 70d7d7abb282404081eb7dc7a644882d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:19.304085 7342 raft_consensus.cc:2454] T 3f405483ce8b4cfea16c161596b64152 P 7b46d70349f041eb928c6a148933aa42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9ef73f81349480f9151fc0b7c37cb48 in term 1.
I20260620 07:09:19.304086 7209 raft_consensus.cc:2454] T 3f405483ce8b4cfea16c161596b64152 P 70d7d7abb282404081eb7dc7a644882d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9ef73f81349480f9151fc0b7c37cb48 in term 1.
I20260620 07:09:19.304467 7011 leader_election.cc:302] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70d7d7abb282404081eb7dc7a644882d, f9ef73f81349480f9151fc0b7c37cb48; no voters:
I20260620 07:09:19.304793 7412 raft_consensus.cc:2781] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:19.305332 7412 raft_consensus.cc:686] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [term 1 LEADER]: Becoming Leader. State: Replica: f9ef73f81349480f9151fc0b7c37cb48, State: Running, Role: LEADER
I20260620 07:09:19.305855 7412 consensus_queue.cc:237] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ef73f81349480f9151fc0b7c37cb48" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 40079 } } peers { permanent_uuid: "70d7d7abb282404081eb7dc7a644882d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 40599 } } peers { permanent_uuid: "7b46d70349f041eb928c6a148933aa42" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35165 } }
I20260620 07:09:19.310045 6930 catalog_manager.cc:5495] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9ef73f81349480f9151fc0b7c37cb48 (127.6.53.65). New cstate: current_term: 1 leader_uuid: "f9ef73f81349480f9151fc0b7c37cb48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ef73f81349480f9151fc0b7c37cb48" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 40079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70d7d7abb282404081eb7dc7a644882d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 40599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b46d70349f041eb928c6a148933aa42" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35165 } health_report { overall_health: UNKNOWN } } }
Jun 20 07:09:19 dist-test-slave-4n6h krb5kdc[6886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:09:19.342489 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.327372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.6.53.65:40079)
0620 07:09:19.327630 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:19.327639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:19.327992 (+ 353us) server_negotiation.cc:407] Connection header received
0620 07:09:19.328125 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:19.328129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:19.328210 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:19.328297 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:19.329767 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.330636 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.332066 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.332487 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.332600 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.336750 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:19.336764 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:19.336779 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:19.336812 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:19.339968 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:19.340560 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:19.340568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:19.340570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:19.340823 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:19.341143 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:19.341147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:19.341149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:19.341488 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:19.341742 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:19.341890 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 07:09:19.342123 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 07:09:19 dist-test-slave-4n6h krb5kdc[6886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939359, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:09:19 dist-test-slave-4n6h krb5kdc[6886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939359, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:09:19.387897 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.374004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.6.53.65:40079)
0620 07:09:19.374223 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:19.374233 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:19.374593 (+ 360us) server_negotiation.cc:407] Connection header received
0620 07:09:19.374768 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:19.374772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:19.374837 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:19.374929 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:19.376193 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.376980 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.378425 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.378845 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.378945 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.382836 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:19.382840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:19.382843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:19.382874 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:19.385461 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:19.386042 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:19.386052 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:19.386055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:19.386213 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:19.386683 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:19.386687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:19.386689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:19.386922 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:19.387132 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:19.387359 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 07:09:19.387522 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 07:09:19.400199 7388 tablet.cc:2369] T 3f405483ce8b4cfea16c161596b64152 P 7b46d70349f041eb928c6a148933aa42: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 07:09:19 dist-test-slave-4n6h krb5kdc[6886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260620 07:09:19.418339 7255 tablet.cc:2369] T 3f405483ce8b4cfea16c161596b64152 P 70d7d7abb282404081eb7dc7a644882d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 07:09:19 dist-test-slave-4n6h krb5kdc[6886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:09:19.430455 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.417155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58278 (local address 127.6.53.65:40079)
0620 07:09:19.417398 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:19.417406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:19.417671 (+ 265us) server_negotiation.cc:407] Connection header received
0620 07:09:19.417914 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:19.417920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:19.418037 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:19.418145 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:19.419506 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.420308 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.421476 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.421881 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.421978 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.425628 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:19.425633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:19.425636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:19.425666 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:19.428172 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:19.428764 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:19.428768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:19.428770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:19.428894 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:19.429186 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:19.429189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:19.429191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:19.429411 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:19.429595 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:19.429864 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 07:09:19.430014 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 07:09:19.432132 7035 tablet_service.cc:2908] Rejecting scan request for tablet 3f405483ce8b4cfea16c161596b64152: Uninitialized: clean time has not yet been initialized
I20260620 07:09:19.435113 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 6992
W20260620 07:09:19.440068 7121 tablet.cc:2369] T 3f405483ce8b4cfea16c161596b64152 P f9ef73f81349480f9151fc0b7c37cb48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:09:19.448549 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.436593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.6.53.65:40079)
0620 07:09:19.436749 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:19.436753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:19.437105 (+ 352us) server_negotiation.cc:407] Connection header received
0620 07:09:19.437228 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:19.437231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:19.437283 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:19.437357 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:19.438523 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.439219 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.440774 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.441258 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.441349 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.443392 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:19.443397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:19.443400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:19.443428 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:19.445968 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:19.446791 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:19.446799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:19.446801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:19.446921 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:19.447306 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:19.447310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:19.447312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:19.447581 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:19.447777 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:19.448089 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 07:09:19.448291 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:09:19.449214 7115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:19.538622 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 6992
I20260620 07:09:19.556265 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 7126
Jun 20 07:09:19 dist-test-slave-4n6h krb5kdc[6886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:09:19.571444 7402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.557018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.6.53.66:40599)
0620 07:09:19.557211 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:19.557215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:19.557235 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:09:19.557293 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:19.557297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:19.557371 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:19.557464 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:19.558708 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.559691 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.560908 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.561316 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.561427 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.565697 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:19.565716 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:19.565731 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:19.565764 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:19.568702 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:19.569313 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:19.569323 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:19.569325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:19.569476 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:19.569866 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:19.569870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:19.569872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:19.570292 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:19.570777 (+ 485us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:19.570998 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 07:09:19.571165 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:09:19.572312 7249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:19.658802 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 7126
I20260620 07:09:19.675393 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 7259
Jun 20 07:09:19 dist-test-slave-4n6h krb5kdc[6886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:09:19.689877 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:19.676178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.6.53.67:35165)
0620 07:09:19.676361 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:19.676365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:19.676425 (+ 60us) server_negotiation.cc:407] Connection header received
0620 07:09:19.676577 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:19.676581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:19.676641 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:19.676724 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:19.678039 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.678912 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.680179 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:19.680579 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:19.680670 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:19.684817 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:19.684849 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:19.684865 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:19.684902 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:19.687512 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:19.688200 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:19.688207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:19.688209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:19.688340 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:19.688637 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:19.688640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:19.688642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:19.688988 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:19.689187 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:19.689406 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 07:09:19.689567 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:09:19.690639 7382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:19.777598 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 7259
I20260620 07:09:19.793625 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 6899
I20260620 07:09:19.794943 6961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:19.904608 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 6899
2026-06-20T07:09:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4027 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:09:19 dist-test-slave-4n6h krb5kdc[7435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:09:19 dist-test-slave-4n6h krb5kdc[7435](info): set up 2 sockets
Jun 20 07:09:19 dist-test-slave-4n6h krb5kdc[7435](info): commencing operation
krb5kdc: starting...
W20260620 07:09:20.982234 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:09:21 dist-test-slave-4n6h krb5kdc[7435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:09:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:09:21.163308 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:42967
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:42967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:21.480094 7448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:21.480438 7448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:21.480595 7448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:21.488008 7448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:21.488147 7448 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 07:09:21.488241 7448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:21.488358 7448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:21.488451 7448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:21.488538 7448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:21.498426 7448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:42967
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:42967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:21.499686 7448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:21.501030 7448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:21.512840 7453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:21.512743 7456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:21.512785 7454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:21.513715 7448 server_base.cc:1034] running on GCE node
I20260620 07:09:21.514557 7448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:21.515897 7448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:21.517150 7448 hybrid_clock.cc:648] HybridClock initialized: now 1781939361517088 us; error 55 us; skew 500 ppm
I20260620 07:09:21.517486 7448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:21 dist-test-slave-4n6h krb5kdc[7435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939361, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:21.520665 7448 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:09:21.522136 7448 webserver.cc:460] Webserver started at http://127.6.53.126:39767/ using document root <none> and password file <none>
I20260620 07:09:21.522871 7448 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:21.523002 7448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:21.523320 7448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:21.526196 7448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "51609e677ec04dcc9d19c32001cdb9a2"
format_stamp: "Formatted at 2026-06-20 07:09:21 on dist-test-slave-4n6h"
server_key: "885ad94d644668072d957e7da530bb7f"
server_key_iv: "ac1d2aab9ace4b8f3ddf541c32cb6526"
server_key_version: "encryptionkey@0"
I20260620 07:09:21.527046 7448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "51609e677ec04dcc9d19c32001cdb9a2"
format_stamp: "Formatted at 2026-06-20 07:09:21 on dist-test-slave-4n6h"
server_key: "885ad94d644668072d957e7da530bb7f"
server_key_iv: "ac1d2aab9ace4b8f3ddf541c32cb6526"
server_key_version: "encryptionkey@0"
I20260620 07:09:21.532022 7448 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:09:21.535288 7463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:21.536553 7448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:21.536829 7448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "51609e677ec04dcc9d19c32001cdb9a2"
format_stamp: "Formatted at 2026-06-20 07:09:21 on dist-test-slave-4n6h"
server_key: "885ad94d644668072d957e7da530bb7f"
server_key_iv: "ac1d2aab9ace4b8f3ddf541c32cb6526"
server_key_version: "encryptionkey@0"
I20260620 07:09:21.537045 7448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:21.551821 7448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:21.555593 7448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:21.555856 7448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:21.578152 7448 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:42967
I20260620 07:09:21.578207 7524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:42967 every 8 connection(s)
I20260620 07:09:21.579636 7448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:21.583428 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:21.589488 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 7448
I20260620 07:09:21.589702 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:09:21.590412 6357 external_mini_cluster.cc:1428] Setting key a270f3674e6c422d07bf54578f1a9155
I20260620 07:09:21.593950 7525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2: Bootstrap starting.
I20260620 07:09:21.596988 7525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:21.598078 7525 log.cc:824] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:21.600718 7525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2: No bootstrap required, opened a new log
Jun 20 07:09:21 dist-test-slave-4n6h krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:21.609805 7525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51609e677ec04dcc9d19c32001cdb9a2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42967 } }
I20260620 07:09:21.610199 7525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:21.610371 7525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51609e677ec04dcc9d19c32001cdb9a2, State: Initialized, Role: FOLLOWER
I20260620 07:09:21.611119 7525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51609e677ec04dcc9d19c32001cdb9a2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42967 } }
I20260620 07:09:21.611596 7525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:09:21.611764 7525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:09:21.611928 7525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:21.613513 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:21.592383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58874 (local address 127.6.53.126:42967)
0620 07:09:21.593152 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:21.593165 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:21.593203 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:09:21.595009 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:21.595030 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:21.595414 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:21.595788 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:21.597164 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:21.598536 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:21.599744 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:21.600381 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:21.600581 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:21.605086 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:21.605093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:21.605107 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:21.605144 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:21.608334 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:21.609176 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:21.609183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:21.609189 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:21.609347 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:21.609739 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:21.609742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:21.609744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:21.610367 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:21.610662 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:21.611077 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 07:09:21.612459 (+ 1382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":351,"threads_started":1}
I20260620 07:09:21.614571 7525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51609e677ec04dcc9d19c32001cdb9a2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42967 } }
I20260620 07:09:21.615067 7525 leader_election.cc:302] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51609e677ec04dcc9d19c32001cdb9a2; no voters:
I20260620 07:09:21.615737 7525 leader_election.cc:288] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:09:21.615859 7530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:21.616739 7530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [term 1 LEADER]: Becoming Leader. State: Replica: 51609e677ec04dcc9d19c32001cdb9a2, State: Running, Role: LEADER
I20260620 07:09:21.617274 7530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51609e677ec04dcc9d19c32001cdb9a2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42967 } }
I20260620 07:09:21.617888 7525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:21.620714 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51609e677ec04dcc9d19c32001cdb9a2. Latest consensus state: current_term: 1 leader_uuid: "51609e677ec04dcc9d19c32001cdb9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51609e677ec04dcc9d19c32001cdb9a2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42967 } } }
I20260620 07:09:21.620740 7531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51609e677ec04dcc9d19c32001cdb9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51609e677ec04dcc9d19c32001cdb9a2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42967 } } }
I20260620 07:09:21.621255 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:21.621300 7531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:21.622898 7537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:09:21.628178 7537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:09:21.637576 7537 catalog_manager.cc:1347] Generated new cluster ID: 482b121896c140f1a02e7b5504a4141c
I20260620 07:09:21.637701 7537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:21.653271 7537 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:21.654650 7537 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:21.661051 7537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2: Generated new TSK 0
I20260620 07:09:21.661852 7537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:09:21.709831 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42967
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:22.076009 7551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:22.076268 7551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:22.076437 7551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:22.083526 7551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:22.083637 7551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:22.083810 7551 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:09:22.083907 7551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:09:22.093598 7551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.53.126:42967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:22.094929 7551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:22.096233 7551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:22.109704 7556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:22.109822 7559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:22.109814 7557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:22.110726 7551 server_base.cc:1034] running on GCE node
I20260620 07:09:22.111339 7551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:22.112260 7551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:22.113516 7551 hybrid_clock.cc:648] HybridClock initialized: now 1781939362113451 us; error 41 us; skew 500 ppm
I20260620 07:09:22.113829 7551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:22 dist-test-slave-4n6h krb5kdc[7435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939362, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:22.117287 7551 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:09:22.119082 7551 webserver.cc:460] Webserver started at http://127.6.53.65:37511/ using document root <none> and password file <none>
I20260620 07:09:22.119776 7551 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:22.119918 7551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:22.120249 7551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:22.123178 7551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "64b78ef3d1954520834dd159c7fefd6e"
format_stamp: "Formatted at 2026-06-20 07:09:22 on dist-test-slave-4n6h"
server_key: "252f6e9d25f2777a39ba2a8203aee8f5"
server_key_iv: "b0a2d1641b63152c7f0ac501759f9dd4"
server_key_version: "encryptionkey@0"
I20260620 07:09:22.123963 7551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "64b78ef3d1954520834dd159c7fefd6e"
format_stamp: "Formatted at 2026-06-20 07:09:22 on dist-test-slave-4n6h"
server_key: "252f6e9d25f2777a39ba2a8203aee8f5"
server_key_iv: "b0a2d1641b63152c7f0ac501759f9dd4"
server_key_version: "encryptionkey@0"
I20260620 07:09:22.129449 7551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:09:22.133347 7566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:22.134907 7551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:09:22.135094 7551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "64b78ef3d1954520834dd159c7fefd6e"
format_stamp: "Formatted at 2026-06-20 07:09:22 on dist-test-slave-4n6h"
server_key: "252f6e9d25f2777a39ba2a8203aee8f5"
server_key_iv: "b0a2d1641b63152c7f0ac501759f9dd4"
server_key_version: "encryptionkey@0"
I20260620 07:09:22.135321 7551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:22.150019 7551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:22.154800 7551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:22.155102 7551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:22.157789 7551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:22.157898 7551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:22.158093 7551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:22.158176 7551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:09:22 dist-test-slave-4n6h krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939362, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:22.182633 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:22.158605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:55015 (local address 127.6.53.126:42967)
0620 07:09:22.161379 (+ 2774us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:22.161386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:22.161413 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:09:22.162744 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:22.162752 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:22.162875 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:22.163041 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:22.165101 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:22.166066 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:22.168801 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:22.169388 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:22.169535 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:22.174970 (+ 5435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:22.174975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:22.174979 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:22.175011 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:22.178333 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:22.179663 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:22.179666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:22.179669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:22.179791 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:22.180426 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:22.180431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:22.180434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:22.180700 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:22.180806 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:22.181984 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0620 07:09:22.182166 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2650,"thread_start_us":2377,"threads_started":1}
I20260620 07:09:22.184739 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:22.159171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42967 (local address 127.6.53.65:55015)
0620 07:09:22.160121 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:22.160166 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:22.162371 (+ 2205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:22.163294 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:22.163309 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:22.164036 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:22.164897 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:22.164927 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:22.166309 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:22.166324 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:22.168598 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:22.168618 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:22.169653 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:22.169661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:22.169876 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:22.170749 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:22.170789 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:22.174753 (+ 3964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:22.178563 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:22.178574 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:22.178599 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:22.179480 (+ 881us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:22.179942 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:22.179951 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:22.179958 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:22.180279 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:22.180844 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:22.180867 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:22.181461 (+ 594us) client_negotiation.cc:769] Sending connection context
0620 07:09:22.181884 (+ 423us) client_negotiation.cc:240] Negotiation successful
0620 07:09:22.182682 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":435,"threads_started":1}
I20260620 07:09:22.206765 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:22.192992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:57473 (local address 127.6.53.126:42967)
0620 07:09:22.193233 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:22.193242 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:22.193873 (+ 631us) server_negotiation.cc:407] Connection header received
0620 07:09:22.194121 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:22.194128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:22.194206 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:22.194364 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:22.195773 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:22.196480 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:22.197767 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:22.198198 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:22.198375 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:22.200374 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:22.200382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:22.200390 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:22.200434 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:22.203509 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:22.204439 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:22.204444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:22.204448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:22.204644 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:22.205250 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:22.205253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:22.205255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:22.205516 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:22.205616 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:22.206362 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 07:09:22.206529 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:09:22.207471 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:22.192915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42967 (local address 127.6.53.65:57473)
0620 07:09:22.193730 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:22.193757 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:22.193975 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:22.194395 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:22.194400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:22.194999 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:22.195627 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:22.195639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:22.196616 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:22.196622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:22.197624 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:22.197634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:22.198454 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:22.198461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:22.198628 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:22.199426 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:22.199449 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:22.200170 (+ 721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:22.203678 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:22.203682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:22.203685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:22.204291 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:22.204817 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:22.204827 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:22.204834 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:22.205091 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:22.205661 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:22.205670 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:22.205978 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 07:09:22.206187 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 07:09:22.206741 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":382,"threads_started":1}
I20260620 07:09:22.217689 7551 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:45473
I20260620 07:09:22.217712 7686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:45473 every 8 connection(s)
I20260620 07:09:22.219024 7551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:09:22.220470 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 7551
I20260620 07:09:22.220712 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:09:22.221343 6357 external_mini_cluster.cc:1428] Setting key 0f0544b70fd85d50139000a82984c2df
I20260620 07:09:22.221123 7687 heartbeater.cc:347] Connected to a master server at 127.6.53.126:42967
I20260620 07:09:22.222327 7687 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:22.223037 7687 heartbeater.cc:511] Master 127.6.53.126:42967 requested a full tablet report, sending...
I20260620 07:09:22.224962 7480 ts_manager.cc:194] Registered new tserver with Master: 64b78ef3d1954520834dd159c7fefd6e (127.6.53.65:45473)
I20260620 07:09:22.226504 7480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:55015
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:09:22.261132 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42967
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:22.577237 7691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:22.577641 7691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:22.577797 7691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:22.589164 7691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:22.589293 7691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:22.589425 7691 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:09:22.589538 7691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:09:22.601095 7691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.53.126:42967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:22.602300 7691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:22.603595 7691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:22.616457 7696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:22.616509 7697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:22.616931 7699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:22.617385 7691 server_base.cc:1034] running on GCE node
I20260620 07:09:22.617978 7691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:22.619074 7691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:22.620339 7691 hybrid_clock.cc:648] HybridClock initialized: now 1781939362620232 us; error 73 us; skew 500 ppm
I20260620 07:09:22.620716 7691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:22 dist-test-slave-4n6h krb5kdc[7435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939362, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:22.630802 7691 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:09:22.632377 7691 webserver.cc:460] Webserver started at http://127.6.53.66:35975/ using document root <none> and password file <none>
I20260620 07:09:22.633078 7691 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:22.633191 7691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:22.633574 7691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:22.636452 7691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "901ec4eb4d344f35a06e246605a81ba8"
format_stamp: "Formatted at 2026-06-20 07:09:22 on dist-test-slave-4n6h"
server_key: "09c38b81d1580a899751a698270eaa7c"
server_key_iv: "8e37be7970c5febac2392d39b57b5ec8"
server_key_version: "encryptionkey@0"
I20260620 07:09:22.637228 7691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "901ec4eb4d344f35a06e246605a81ba8"
format_stamp: "Formatted at 2026-06-20 07:09:22 on dist-test-slave-4n6h"
server_key: "09c38b81d1580a899751a698270eaa7c"
server_key_iv: "8e37be7970c5febac2392d39b57b5ec8"
server_key_version: "encryptionkey@0"
I20260620 07:09:22.643354 7691 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260620 07:09:22.647610 7706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:22.649039 7691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:09:22.649173 7691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "901ec4eb4d344f35a06e246605a81ba8"
format_stamp: "Formatted at 2026-06-20 07:09:22 on dist-test-slave-4n6h"
server_key: "09c38b81d1580a899751a698270eaa7c"
server_key_iv: "8e37be7970c5febac2392d39b57b5ec8"
server_key_version: "encryptionkey@0"
I20260620 07:09:22.649341 7691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:22.668931 7691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:22.672833 7691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:22.673051 7691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:22.675740 7691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:22.675837 7691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:22.676033 7691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:22.676108 7691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:09:22 dist-test-slave-4n6h krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939362, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:22.700881 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:22.676152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:48403 (local address 127.6.53.126:42967)
0620 07:09:22.676345 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:22.676352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:22.677720 (+ 1368us) server_negotiation.cc:407] Connection header received
0620 07:09:22.680007 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:22.680016 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:22.680133 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:22.680251 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:22.682550 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:22.683642 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:22.686270 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:22.687073 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:22.687272 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:22.692567 (+ 5295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:22.692576 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:22.692584 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:22.692638 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:22.696892 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:22.698001 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:22.698010 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:22.698014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:22.698157 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:22.698844 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:22.698850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:22.698854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:22.699317 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:22.699493 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:22.700221 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 07:09:22.700455 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 07:09:22.702023 7715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:22.676677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42967 (local address 127.6.53.66:48403)
0620 07:09:22.677539 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:22.677592 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:22.679689 (+ 2097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:22.680517 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:22.680524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:22.681218 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:22.682284 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:22.682318 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:22.683805 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:22.683813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:22.686085 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:22.686104 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:22.687366 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:22.687380 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:22.687553 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:22.688336 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:22.688384 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:22.692287 (+ 3903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:22.697082 (+ 4795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:22.697094 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:22.697123 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:22.697816 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:22.698381 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:22.698390 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:22.698395 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:22.698701 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:22.699472 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:22.699480 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:22.699840 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 07:09:22.700130 (+ 290us) client_negotiation.cc:240] Negotiation successful
0620 07:09:22.700662 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":366,"threads_started":1}
I20260620 07:09:22.724529 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:22.708711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:48989 (local address 127.6.53.126:42967)
0620 07:09:22.708886 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:22.708890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:22.709688 (+ 798us) server_negotiation.cc:407] Connection header received
0620 07:09:22.709834 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:22.709842 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:22.709968 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:22.710106 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:22.711531 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:22.712894 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:22.714232 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:22.715126 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:22.715280 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:22.717020 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:22.717029 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:22.717037 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:22.717085 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:22.721441 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:22.722144 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:22.722152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:22.722158 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:22.722393 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:22.722858 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:22.722865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:22.722868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:22.723228 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:22.723401 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:22.723894 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 07:09:22.724122 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:09:22.724666 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:22.708786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42967 (local address 127.6.53.66:48989)
0620 07:09:22.709568 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:22.709589 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:22.709719 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:22.710147 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:22.710151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:22.710833 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:22.711353 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:22.711372 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:22.713078 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:22.713082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:22.714102 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:22.714113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:22.715306 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:22.715311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:22.715427 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:22.716118 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:22.716134 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:22.716816 (+ 682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:22.721617 (+ 4801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:22.721622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:22.721625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:22.722024 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:22.722539 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:22.722547 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:22.722560 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:22.722728 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:22.723381 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:22.723387 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:22.723597 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 07:09:22.723785 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 07:09:22.724155 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":447,"threads_started":1}
I20260620 07:09:22.734311 7691 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:44793
I20260620 07:09:22.734350 7825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:44793 every 8 connection(s)
I20260620 07:09:22.735468 7691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:09:22.737416 7826 heartbeater.cc:347] Connected to a master server at 127.6.53.126:42967
I20260620 07:09:22.737660 7826 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:22.738405 7826 heartbeater.cc:511] Master 127.6.53.126:42967 requested a full tablet report, sending...
I20260620 07:09:22.738664 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 7691
I20260620 07:09:22.738850 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:09:22.739378 6357 external_mini_cluster.cc:1428] Setting key 23e9a1abfb7220a3bd7b8cb20d248056
I20260620 07:09:22.739961 7480 ts_manager.cc:194] Registered new tserver with Master: 901ec4eb4d344f35a06e246605a81ba8 (127.6.53.66:44793)
I20260620 07:09:22.741016 7480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:48403
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:09:22.778012 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42967
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:42077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:23.090241 7830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:23.090600 7830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:23.090795 7830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:23.097792 7830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:23.097939 7830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:23.098095 7830 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:09:23.098235 7830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:09:23.108294 7830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:42077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.53.126:42967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:23.109570 7830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:23.110991 7830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:23.123935 7836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:23.123991 7835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:23.123965 7838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:23.124722 7830 server_base.cc:1034] running on GCE node
I20260620 07:09:23.125339 7830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:23.126307 7830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:23.127527 7830 hybrid_clock.cc:648] HybridClock initialized: now 1781939363127460 us; error 55 us; skew 500 ppm
I20260620 07:09:23.127900 7830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:23 dist-test-slave-4n6h krb5kdc[7435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939363, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:23.131071 7830 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:09:23.132455 7830 webserver.cc:460] Webserver started at http://127.6.53.67:42865/ using document root <none> and password file <none>
I20260620 07:09:23.133118 7830 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:23.133245 7830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:23.133549 7830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:23.136372 7830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "7202953660a745a5b0476d4e277ae745"
format_stamp: "Formatted at 2026-06-20 07:09:23 on dist-test-slave-4n6h"
server_key: "9114fb489b31751d65a163f7e869abe4"
server_key_iv: "8e77b8b608402ccffb39967992576274"
server_key_version: "encryptionkey@0"
I20260620 07:09:23.137141 7830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "7202953660a745a5b0476d4e277ae745"
format_stamp: "Formatted at 2026-06-20 07:09:23 on dist-test-slave-4n6h"
server_key: "9114fb489b31751d65a163f7e869abe4"
server_key_iv: "8e77b8b608402ccffb39967992576274"
server_key_version: "encryptionkey@0"
I20260620 07:09:23.142342 7830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 07:09:23.145551 7845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:23.147190 7830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:23.147342 7830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "7202953660a745a5b0476d4e277ae745"
format_stamp: "Formatted at 2026-06-20 07:09:23 on dist-test-slave-4n6h"
server_key: "9114fb489b31751d65a163f7e869abe4"
server_key_iv: "8e77b8b608402ccffb39967992576274"
server_key_version: "encryptionkey@0"
I20260620 07:09:23.147552 7830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:23.165200 7830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:23.169544 7830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:23.169847 7830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:23.172817 7830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:23.173031 7830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 07:09:23.173208 7830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:23.173316 7830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:09:23 dist-test-slave-4n6h krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939363, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:23.195555 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.173154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:40787 (local address 127.6.53.126:42967)
0620 07:09:23.173341 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:23.173347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:23.174779 (+ 1432us) server_negotiation.cc:407] Connection header received
0620 07:09:23.176764 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:23.176770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:23.176862 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:23.176990 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:23.178855 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.180321 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.182811 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.183500 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.183642 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.187066 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:23.187071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:23.187074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:23.187154 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:23.191174 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:23.192327 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:23.192330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:23.192332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:23.192455 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:23.193096 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:23.193102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:23.193107 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:23.193391 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:23.193580 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:23.194705 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0620 07:09:23.195004 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:09:23.196415 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.173653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42967 (local address 127.6.53.67:40787)
0620 07:09:23.174580 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:23.174628 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:23.176493 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:23.177221 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:23.177229 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:23.177720 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:23.178597 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.178653 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.180534 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.180546 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.182672 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.182684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.183733 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.183741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.183963 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.184604 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:23.184624 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:23.186834 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:23.191338 (+ 4504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:23.191352 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:23.191374 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:23.192184 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:23.192579 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:23.192587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:23.192593 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:23.192943 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:23.193686 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:23.193697 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:23.194130 (+ 433us) client_negotiation.cc:769] Sending connection context
0620 07:09:23.194528 (+ 398us) client_negotiation.cc:240] Negotiation successful
0620 07:09:23.195174 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":401,"threads_started":1}
I20260620 07:09:23.217880 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.202565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42967 (local address 127.6.53.67:46291)
0620 07:09:23.203425 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:23.203441 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:23.203709 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:23.204269 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:23.204274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:23.205024 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:23.205485 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.205498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.206832 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.206837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.207825 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.207836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.208568 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.208574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.208756 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.209378 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:23.209395 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:23.210067 (+ 672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:23.214609 (+ 4542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:23.214618 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:23.214623 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:23.215391 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:23.215876 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:23.215883 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:23.215887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:23.216048 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:23.216741 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:23.216747 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:23.216956 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 07:09:23.217171 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 07:09:23.217448 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":471,"threads_started":1}
I20260620 07:09:23.218961 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.202506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:46291 (local address 127.6.53.126:42967)
0620 07:09:23.202704 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:23.202713 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:23.203548 (+ 835us) server_negotiation.cc:407] Connection header received
0620 07:09:23.203878 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:23.203883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:23.203972 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:23.204074 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:23.205638 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.206683 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.207974 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.208376 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.208524 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.210292 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:23.210300 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:23.210304 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:23.210349 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:23.214459 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:23.215556 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:23.215565 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:23.215571 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:23.215760 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:23.216256 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:23.216264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:23.216269 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:23.216601 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:23.216774 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:23.218448 (+ 1674us) server_negotiation.cc:299] Negotiation successful
0620 07:09:23.218630 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:09:23.229179 7687 heartbeater.cc:503] Master 127.6.53.126:42967 was elected leader, sending a full tablet report...
I20260620 07:09:23.237394 7830 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:37175
I20260620 07:09:23.237417 7964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:37175 every 8 connection(s)
I20260620 07:09:23.238788 7830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:09:23.240984 7965 heartbeater.cc:347] Connected to a master server at 127.6.53.126:42967
I20260620 07:09:23.241207 7965 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:23.241895 7965 heartbeater.cc:511] Master 127.6.53.126:42967 requested a full tablet report, sending...
I20260620 07:09:23.243229 7480 ts_manager.cc:194] Registered new tserver with Master: 7202953660a745a5b0476d4e277ae745 (127.6.53.67:37175)
I20260620 07:09:23.244107 7480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:40787
I20260620 07:09:23.244818 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 7830
I20260620 07:09:23.245050 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:09:23.245508 6357 external_mini_cluster.cc:1428] Setting key bb3ed162b11b5f374f8b49ddc24381ce
I20260620 07:09:23.248327 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 07:09:23 dist-test-slave-4n6h krb5kdc[7435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939363, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:09:23 dist-test-slave-4n6h krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939363, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:23.293284 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.278886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.6.53.126:42967)
0620 07:09:23.279067 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:23.279071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:23.279401 (+ 330us) server_negotiation.cc:407] Connection header received
0620 07:09:23.279592 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:23.279597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:23.279666 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:23.279771 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:23.281288 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.282110 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.283464 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.283877 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.284014 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.287870 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:23.287875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:23.287878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:23.287913 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:23.290568 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:23.291289 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:23.291296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:23.291299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:23.291440 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:23.291826 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:23.291835 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:23.291837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:23.292146 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:23.292402 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:23.292696 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 07:09:23.292918 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:09:23.298018 7480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:09:23.305147 7480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:09:23.335736 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.318061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42392 (local address 127.6.53.67:37175)
0620 07:09:23.319036 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:23.319048 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:23.325726 (+ 6678us) server_negotiation.cc:407] Connection header received
0620 07:09:23.327349 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:23.327360 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:23.327621 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:23.327863 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:23.329501 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.330856 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.332979 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.334272 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.334403 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.334649 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:23.335223 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 07:09:23.335407 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":535,"threads_started":1}
I20260620 07:09:23.335740 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.316933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:44793 (local address 127.0.0.1:38960)
0620 07:09:23.323047 (+ 6114us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:23.323095 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:23.323304 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:23.327857 (+ 4553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:23.327864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:23.327898 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:23.328480 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.328492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.330289 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.330296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.332894 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.332914 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.334738 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.334743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.334870 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.334904 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 07:09:23.334992 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 07:09:23.335224 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5741,"mutex_wait_us":1,"thread_start_us":6495,"threads_started":1}
I20260620 07:09:23.336108 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.317066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.6.53.66:44793)
0620 07:09:23.327068 (+ 10002us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:23.327076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:23.327103 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:09:23.327296 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:23.327310 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:23.327632 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:23.327843 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:23.328708 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.330112 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.333089 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.334523 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.335384 (+ 861us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.335591 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:23.335655 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 07:09:23.335832 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9858,"thread_start_us":9746,"threads_started":1}
I20260620 07:09:23.336294 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.323836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:45473 (local address 127.0.0.1:54034)
0620 07:09:23.327301 (+ 3465us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:23.327318 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:23.327444 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:23.328069 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:23.328073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:23.328100 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:23.329290 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.329303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.331118 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.331122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.333391 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.333404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.335466 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.335472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.335647 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.335690 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 07:09:23.335787 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 07:09:23.335906 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3360,"mutex_wait_us":91,"thread_start_us":3170,"threads_started":1}
I20260620 07:09:23.336644 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.317276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:37175 (local address 127.0.0.1:42392)
0620 07:09:23.323765 (+ 6489us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:23.323789 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:23.327144 (+ 3355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:23.328224 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:23.328231 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:23.328258 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:23.329247 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.329264 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.330989 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.330996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.332820 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.332832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.334805 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.334812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.334997 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.335063 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 07:09:23.335141 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 07:09:23.335267 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6330,"mutex_wait_us":103,"thread_start_us":5220,"threads_started":1}
I20260620 07:09:23.336880 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.324085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.6.53.65:45473)
0620 07:09:23.326815 (+ 2730us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:23.326828 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:23.327421 (+ 593us) server_negotiation.cc:407] Connection header received
0620 07:09:23.327589 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:23.327596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:23.327795 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:23.327957 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:23.329500 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.330909 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.333547 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.334981 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.335119 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.335309 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:23.336402 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0620 07:09:23.336580 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2520,"thread_start_us":2356,"threads_started":1}
I20260620 07:09:23.339799 7756 tablet_service.cc:1467] Processing CreateTablet for tablet 7daf4bbaa12448ccaf61403f92bd2f01 (DEFAULT_TABLE table=test-table [id=8bdd27aee2554731b0469a28cb0f56c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:23.340152 7617 tablet_service.cc:1467] Processing CreateTablet for tablet 7daf4bbaa12448ccaf61403f92bd2f01 (DEFAULT_TABLE table=test-table [id=8bdd27aee2554731b0469a28cb0f56c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:23.341209 7756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7daf4bbaa12448ccaf61403f92bd2f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:23.341451 7617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7daf4bbaa12448ccaf61403f92bd2f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:23.348187 7895 tablet_service.cc:1467] Processing CreateTablet for tablet 7daf4bbaa12448ccaf61403f92bd2f01 (DEFAULT_TABLE table=test-table [id=8bdd27aee2554731b0469a28cb0f56c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:23.349673 7895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7daf4bbaa12448ccaf61403f92bd2f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:23.352808 7984 tablet_bootstrap.cc:492] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8: Bootstrap starting.
I20260620 07:09:23.354892 7985 tablet_bootstrap.cc:492] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e: Bootstrap starting.
I20260620 07:09:23.357270 7984 tablet_bootstrap.cc:654] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:23.358311 7985 tablet_bootstrap.cc:654] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:23.358593 7984 log.cc:824] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:23.359508 7985 log.cc:824] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:23.361598 7984 tablet_bootstrap.cc:492] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8: No bootstrap required, opened a new log
I20260620 07:09:23.361883 7984 ts_tablet_manager.cc:1397] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260620 07:09:23.362401 7985 tablet_bootstrap.cc:492] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e: No bootstrap required, opened a new log
I20260620 07:09:23.362665 7985 ts_tablet_manager.cc:1397] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 07:09:23.366968 7988 tablet_bootstrap.cc:492] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745: Bootstrap starting.
I20260620 07:09:23.370512 7988 tablet_bootstrap.cc:654] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:23.371037 7985 raft_consensus.cc:348] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.371382 7985 raft_consensus.cc:374] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:23.371507 7985 raft_consensus.cc:729] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b78ef3d1954520834dd159c7fefd6e, State: Initialized, Role: FOLLOWER
I20260620 07:09:23.371714 7988 log.cc:824] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:23.372114 7985 consensus_queue.cc:260] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.373730 7985 ts_tablet_manager.cc:1428] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 07:09:23.373956 7984 raft_consensus.cc:348] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.374420 7984 raft_consensus.cc:374] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:23.374527 7984 raft_consensus.cc:729] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901ec4eb4d344f35a06e246605a81ba8, State: Initialized, Role: FOLLOWER
I20260620 07:09:23.374984 7988 tablet_bootstrap.cc:492] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745: No bootstrap required, opened a new log
I20260620 07:09:23.375270 7988 ts_tablet_manager.cc:1397] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 07:09:23.375118 7984 consensus_queue.cc:260] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.376184 7826 heartbeater.cc:503] Master 127.6.53.126:42967 was elected leader, sending a full tablet report...
I20260620 07:09:23.376817 7984 ts_tablet_manager.cc:1428] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260620 07:09:23.383725 7988 raft_consensus.cc:348] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.384096 7988 raft_consensus.cc:374] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:23.384217 7988 raft_consensus.cc:729] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7202953660a745a5b0476d4e277ae745, State: Initialized, Role: FOLLOWER
I20260620 07:09:23.384809 7988 consensus_queue.cc:260] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.385809 7965 heartbeater.cc:503] Master 127.6.53.126:42967 was elected leader, sending a full tablet report...
I20260620 07:09:23.386229 7988 ts_tablet_manager.cc:1428] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260620 07:09:23.471490 7688 tablet.cc:2369] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:09:23.487471 7827 tablet.cc:2369] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:09:23.490628 7966 tablet.cc:2369] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:09:23.656217 7991 raft_consensus.cc:482] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:09:23.656497 7991 raft_consensus.cc:504] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.658165 7991 leader_election.cc:288] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64b78ef3d1954520834dd159c7fefd6e (127.6.53.65:45473), 7202953660a745a5b0476d4e277ae745 (127.6.53.67:37175)
I20260620 07:09:23.665622 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.658617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:37743 (local address 127.6.53.65:45473)
0620 07:09:23.658829 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:23.658837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:23.659911 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 07:09:23.660153 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:23.660158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:23.660227 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:23.660323 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:23.660924 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.661658 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.663827 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.664607 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.664773 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.664890 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:23.665233 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 07:09:23.665360 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:09:23.665648 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.658962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:52073 (local address 127.6.53.67:37175)
0620 07:09:23.659160 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:23.659168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:23.660118 (+ 950us) server_negotiation.cc:407] Connection header received
0620 07:09:23.660325 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:23.660329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:23.660387 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:23.660488 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:23.661123 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.661799 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.663830 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.664634 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.664788 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.664910 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:23.665265 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 07:09:23.665395 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 07:09:23.665799 7993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.658480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:45473 (local address 127.6.53.66:37743)
0620 07:09:23.659782 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:23.659799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:23.659995 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:23.660352 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:23.660356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:23.660381 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:23.660771 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.660780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.661788 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.661796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.663655 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.663667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.664868 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.664877 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.665057 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.665096 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 07:09:23.665198 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 07:09:23.665327 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1188,"mutex_wait_us":51,"spinlock_wait_cycles":17664,"thread_start_us":925,"threads_started":1}
I20260620 07:09:23.665911 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.659233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:37175 (local address 127.6.53.66:52073)
0620 07:09:23.660003 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:23.660020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:23.660175 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:23.660505 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:23.660509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:23.660548 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:23.660958 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.660967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.661940 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.661945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.663661 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.663673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.664907 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.664914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.665111 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.665134 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:09:23.665239 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 07:09:23.665364 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":61,"thread_start_us":442,"threads_started":1}
I20260620 07:09:23.666829 7920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7daf4bbaa12448ccaf61403f92bd2f01" candidate_uuid: "901ec4eb4d344f35a06e246605a81ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7202953660a745a5b0476d4e277ae745" is_pre_election: true
I20260620 07:09:23.667128 7638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7daf4bbaa12448ccaf61403f92bd2f01" candidate_uuid: "901ec4eb4d344f35a06e246605a81ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b78ef3d1954520834dd159c7fefd6e" is_pre_election: true
I20260620 07:09:23.667322 7920 raft_consensus.cc:2454] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 901ec4eb4d344f35a06e246605a81ba8 in term 0.
I20260620 07:09:23.667603 7638 raft_consensus.cc:2454] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 901ec4eb4d344f35a06e246605a81ba8 in term 0.
I20260620 07:09:23.668008 7708 leader_election.cc:302] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7202953660a745a5b0476d4e277ae745, 901ec4eb4d344f35a06e246605a81ba8; no voters:
I20260620 07:09:23.668408 7991 raft_consensus.cc:2781] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:09:23.668556 7991 raft_consensus.cc:482] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:09:23.668658 7991 raft_consensus.cc:3037] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:23.670879 7991 raft_consensus.cc:504] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.671464 7991 leader_election.cc:288] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [CANDIDATE]: Term 1 election: Requested vote from peers 64b78ef3d1954520834dd159c7fefd6e (127.6.53.65:45473), 7202953660a745a5b0476d4e277ae745 (127.6.53.67:37175)
I20260620 07:09:23.671962 7638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7daf4bbaa12448ccaf61403f92bd2f01" candidate_uuid: "901ec4eb4d344f35a06e246605a81ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b78ef3d1954520834dd159c7fefd6e"
I20260620 07:09:23.671962 7920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7daf4bbaa12448ccaf61403f92bd2f01" candidate_uuid: "901ec4eb4d344f35a06e246605a81ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7202953660a745a5b0476d4e277ae745"
I20260620 07:09:23.672164 7920 raft_consensus.cc:3037] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:23.672164 7638 raft_consensus.cc:3037] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:23.675343 7638 raft_consensus.cc:2454] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 901ec4eb4d344f35a06e246605a81ba8 in term 1.
I20260620 07:09:23.675347 7920 raft_consensus.cc:2454] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 901ec4eb4d344f35a06e246605a81ba8 in term 1.
I20260620 07:09:23.675817 7709 leader_election.cc:302] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 64b78ef3d1954520834dd159c7fefd6e, 7202953660a745a5b0476d4e277ae745, 901ec4eb4d344f35a06e246605a81ba8; no voters:
I20260620 07:09:23.676218 7991 raft_consensus.cc:2781] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:23.676719 7991 raft_consensus.cc:686] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [term 1 LEADER]: Becoming Leader. State: Replica: 901ec4eb4d344f35a06e246605a81ba8, State: Running, Role: LEADER
I20260620 07:09:23.677224 7991 consensus_queue.cc:237] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.681735 7478 catalog_manager.cc:5495] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 901ec4eb4d344f35a06e246605a81ba8 (127.6.53.66). New cstate: current_term: 1 leader_uuid: "901ec4eb4d344f35a06e246605a81ba8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } health_report { overall_health: UNKNOWN } } }
I20260620 07:09:23.788416 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.783403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.6.53.66:44793)
0620 07:09:23.783593 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:23.783598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:23.783614 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:09:23.783694 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:23.783698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:23.783773 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:23.783890 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:23.784252 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.785004 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.786357 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.786759 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.786850 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.787306 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:23.787886 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:23.787993 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:23.788150 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 07:09:23.788215 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:09:23.796622 7920 raft_consensus.cc:1261] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745 [term 1 FOLLOWER]: Refusing update from remote peer 901ec4eb4d344f35a06e246605a81ba8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:23.796638 7638 raft_consensus.cc:1261] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e [term 1 FOLLOWER]: Refusing update from remote peer 901ec4eb4d344f35a06e246605a81ba8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:23.797609 7995 consensus_queue.cc:1035] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:23.797958 7991 consensus_queue.cc:1035] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:23.811926 8000 mvcc.cc:205] Tried to move back new op lower bound from 7298823634100338688 to 7298823633625858048. Current Snapshot: MvccSnapshot[applied={T|T < 7298823634100338688}]
I20260620 07:09:23.831187 7478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:09:23.831440 7478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:09:23.834982 7478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2: Sending DeleteTablet for 3 replicas of tablet 7daf4bbaa12448ccaf61403f92bd2f01
I20260620 07:09:23.836643 7756 tablet_service.cc:1514] Processing DeleteTablet for tablet 7daf4bbaa12448ccaf61403f92bd2f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:23 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:38960
I20260620 07:09:23.836619 7617 tablet_service.cc:1514] Processing DeleteTablet for tablet 7daf4bbaa12448ccaf61403f92bd2f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:23 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:54034
I20260620 07:09:23.838294 7895 tablet_service.cc:1514] Processing DeleteTablet for tablet 7daf4bbaa12448ccaf61403f92bd2f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:23 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:42392
I20260620 07:09:23.839560 8008 tablet_replica.cc:330] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8: stopping tablet replica
I20260620 07:09:23.839550 7478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 07:09:23.840016 8008 raft_consensus.cc:2229] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [term 1 LEADER]: Raft consensus shutting down.
W20260620 07:09:23.840508 7478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:09:23.840855 8008 raft_consensus.cc:2258] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:23.843076 8008 ts_tablet_manager.cc:1905] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:23.850111 7616 tablet_service.cc:1467] Processing CreateTablet for tablet 98242213c8f24957a87fd423ea2450b4 (DEFAULT_TABLE table=test-table [id=a8813780ca0c43a8b915b8f24aa27192]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:23.850787 7616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98242213c8f24957a87fd423ea2450b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:23.850339 7756 tablet_service.cc:1467] Processing CreateTablet for tablet 98242213c8f24957a87fd423ea2450b4 (DEFAULT_TABLE table=test-table [id=a8813780ca0c43a8b915b8f24aa27192]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:23.851367 7756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98242213c8f24957a87fd423ea2450b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:23.851346 7894 tablet_service.cc:1467] Processing CreateTablet for tablet 98242213c8f24957a87fd423ea2450b4 (DEFAULT_TABLE table=test-table [id=a8813780ca0c43a8b915b8f24aa27192]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:23.851881 7894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98242213c8f24957a87fd423ea2450b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:23.857643 7985 tablet_bootstrap.cc:492] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e: Bootstrap starting.
I20260620 07:09:23.858458 7988 tablet_bootstrap.cc:492] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745: Bootstrap starting.
I20260620 07:09:23.859316 7985 tablet_bootstrap.cc:654] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:23.859867 7984 tablet_bootstrap.cc:492] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8: Bootstrap starting.
I20260620 07:09:23.860059 7988 tablet_bootstrap.cc:654] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:23.861697 7984 tablet_bootstrap.cc:654] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:23.864267 8009 tablet_replica.cc:330] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745: stopping tablet replica
I20260620 07:09:23.864734 8009 raft_consensus.cc:2229] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:09:23.865262 8009 raft_consensus.cc:2258] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:23.868772 8008 ts_tablet_manager.cc:1918] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:09:23.868973 8008 log.cc:1196] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/7daf4bbaa12448ccaf61403f92bd2f01
I20260620 07:09:23.869585 8008 ts_tablet_manager.cc:1939] T 7daf4bbaa12448ccaf61403f92bd2f01 P 901ec4eb4d344f35a06e246605a81ba8: Deleting consensus metadata
I20260620 07:09:23.871368 8010 tablet_replica.cc:330] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e: stopping tablet replica
I20260620 07:09:23.871671 8009 ts_tablet_manager.cc:1905] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:23.871829 8010 raft_consensus.cc:2229] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:09:23.872325 8010 raft_consensus.cc:2258] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:23.873660 7984 tablet_bootstrap.cc:492] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8: No bootstrap required, opened a new log
I20260620 07:09:23.873843 7984 ts_tablet_manager.cc:1397] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260620 07:09:23.874832 7984 raft_consensus.cc:348] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.875115 7984 raft_consensus.cc:374] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:23.875226 7984 raft_consensus.cc:729] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901ec4eb4d344f35a06e246605a81ba8, State: Initialized, Role: FOLLOWER
I20260620 07:09:23.875569 7984 consensus_queue.cc:260] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.876842 7988 tablet_bootstrap.cc:492] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745: No bootstrap required, opened a new log
I20260620 07:09:23.876986 7984 ts_tablet_manager.cc:1428] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:23.877095 7988 ts_tablet_manager.cc:1397] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.006s
I20260620 07:09:23.877358 7985 tablet_bootstrap.cc:492] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e: No bootstrap required, opened a new log
I20260620 07:09:23.877533 7985 ts_tablet_manager.cc:1397] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.004s
I20260620 07:09:23.878402 7985 raft_consensus.cc:348] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.878631 7985 raft_consensus.cc:374] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:23.878747 7985 raft_consensus.cc:729] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b78ef3d1954520834dd159c7fefd6e, State: Initialized, Role: FOLLOWER
I20260620 07:09:23.879082 8010 ts_tablet_manager.cc:1905] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:23.879076 7985 consensus_queue.cc:260] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.879175 7988 raft_consensus.cc:348] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.879532 7988 raft_consensus.cc:374] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:23.879669 7988 raft_consensus.cc:729] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7202953660a745a5b0476d4e277ae745, State: Initialized, Role: FOLLOWER
I20260620 07:09:23.880003 7988 consensus_queue.cc:260] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.880532 7985 ts_tablet_manager.cc:1428] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:23.881428 7988 ts_tablet_manager.cc:1428] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260620 07:09:23.882157 7466 catalog_manager.cc:4841] TS 901ec4eb4d344f35a06e246605a81ba8 (127.6.53.66:44793): tablet 7daf4bbaa12448ccaf61403f92bd2f01 (table test-table [id=8bdd27aee2554731b0469a28cb0f56c6]) successfully deleted
I20260620 07:09:23.886137 8010 ts_tablet_manager.cc:1918] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:09:23.886324 8010 log.cc:1196] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/7daf4bbaa12448ccaf61403f92bd2f01
I20260620 07:09:23.886879 8010 ts_tablet_manager.cc:1939] T 7daf4bbaa12448ccaf61403f92bd2f01 P 64b78ef3d1954520834dd159c7fefd6e: Deleting consensus metadata
I20260620 07:09:23.886849 8009 ts_tablet_manager.cc:1918] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:09:23.887022 8009 log.cc:1196] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/7daf4bbaa12448ccaf61403f92bd2f01
I20260620 07:09:23.887499 8009 ts_tablet_manager.cc:1939] T 7daf4bbaa12448ccaf61403f92bd2f01 P 7202953660a745a5b0476d4e277ae745: Deleting consensus metadata
I20260620 07:09:23.888630 7466 catalog_manager.cc:4841] TS 64b78ef3d1954520834dd159c7fefd6e (127.6.53.65:45473): tablet 7daf4bbaa12448ccaf61403f92bd2f01 (table test-table [id=8bdd27aee2554731b0469a28cb0f56c6]) successfully deleted
I20260620 07:09:23.889112 7465 catalog_manager.cc:4841] TS 7202953660a745a5b0476d4e277ae745 (127.6.53.67:37175): tablet 7daf4bbaa12448ccaf61403f92bd2f01 (table test-table [id=8bdd27aee2554731b0469a28cb0f56c6]) successfully deleted
I20260620 07:09:23.965184 7992 raft_consensus.cc:482] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:09:23.965462 7992 raft_consensus.cc:504] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.966987 7992 leader_election.cc:288] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64b78ef3d1954520834dd159c7fefd6e (127.6.53.65:45473), 901ec4eb4d344f35a06e246605a81ba8 (127.6.53.66:44793)
I20260620 07:09:23.975041 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.967394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:52253 (local address 127.6.53.65:45473)
0620 07:09:23.967622 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:23.967628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:23.967950 (+ 322us) server_negotiation.cc:407] Connection header received
0620 07:09:23.968269 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:23.968277 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:23.968383 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:23.968454 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:23.969204 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.970355 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.973292 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.974094 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.974248 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.974369 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:23.974638 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 07:09:23.974768 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 07:09:23.975395 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.967281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:45473 (local address 127.6.53.67:52253)
0620 07:09:23.967793 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:23.967809 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:23.968081 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:23.968675 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:23.968681 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:23.968713 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:23.969031 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.969042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.970524 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.970529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.973080 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.973096 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.974302 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.974309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.974476 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.974503 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 07:09:23.974611 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 07:09:23.974747 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":179,"thread_start_us":268,"threads_started":1}
I20260620 07:09:23.975858 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.968124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:57455 (local address 127.6.53.66:44793)
0620 07:09:23.968292 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:23.968298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:23.968676 (+ 378us) server_negotiation.cc:407] Connection header received
0620 07:09:23.968826 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:23.968832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:23.968913 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:23.969009 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:23.969549 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.970555 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.973240 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.974560 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.974705 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.974794 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:23.975379 (+ 585us) server_negotiation.cc:299] Negotiation successful
0620 07:09:23.975555 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 07:09:23.975926 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:23.967988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:44793 (local address 127.6.53.67:57455)
0620 07:09:23.968566 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:23.968584 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:23.968695 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:23.969057 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:23.969063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:23.969096 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:23.969400 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.969407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.970707 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.970712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.973102 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:23.973110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:23.974950 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:23.974958 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:23.975169 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:23.975204 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:09:23.975334 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 07:09:23.975516 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":125,"thread_start_us":289,"threads_started":1}
I20260620 07:09:23.976500 7638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98242213c8f24957a87fd423ea2450b4" candidate_uuid: "7202953660a745a5b0476d4e277ae745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b78ef3d1954520834dd159c7fefd6e" is_pre_election: true
I20260620 07:09:23.976785 7638 raft_consensus.cc:2454] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7202953660a745a5b0476d4e277ae745 in term 0.
I20260620 07:09:23.976810 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98242213c8f24957a87fd423ea2450b4" candidate_uuid: "7202953660a745a5b0476d4e277ae745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901ec4eb4d344f35a06e246605a81ba8" is_pre_election: true
I20260620 07:09:23.977265 7780 raft_consensus.cc:2454] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7202953660a745a5b0476d4e277ae745 in term 0.
I20260620 07:09:23.977298 7848 leader_election.cc:302] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b78ef3d1954520834dd159c7fefd6e, 7202953660a745a5b0476d4e277ae745; no voters:
I20260620 07:09:23.977762 7992 raft_consensus.cc:2781] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:09:23.977892 7992 raft_consensus.cc:482] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:09:23.977998 7992 raft_consensus.cc:3037] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:23.981159 7992 raft_consensus.cc:504] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.981835 7992 leader_election.cc:288] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [CANDIDATE]: Term 1 election: Requested vote from peers 64b78ef3d1954520834dd159c7fefd6e (127.6.53.65:45473), 901ec4eb4d344f35a06e246605a81ba8 (127.6.53.66:44793)
I20260620 07:09:23.982291 7638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98242213c8f24957a87fd423ea2450b4" candidate_uuid: "7202953660a745a5b0476d4e277ae745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b78ef3d1954520834dd159c7fefd6e"
I20260620 07:09:23.982398 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98242213c8f24957a87fd423ea2450b4" candidate_uuid: "7202953660a745a5b0476d4e277ae745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901ec4eb4d344f35a06e246605a81ba8"
I20260620 07:09:23.982537 7638 raft_consensus.cc:3037] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:23.982592 7780 raft_consensus.cc:3037] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:23.985502 7638 raft_consensus.cc:2454] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7202953660a745a5b0476d4e277ae745 in term 1.
I20260620 07:09:23.985878 7848 leader_election.cc:302] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b78ef3d1954520834dd159c7fefd6e, 7202953660a745a5b0476d4e277ae745; no voters:
I20260620 07:09:23.985965 7780 raft_consensus.cc:2454] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7202953660a745a5b0476d4e277ae745 in term 1.
I20260620 07:09:23.986192 7992 raft_consensus.cc:2781] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:23.986714 7992 raft_consensus.cc:686] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [term 1 LEADER]: Becoming Leader. State: Replica: 7202953660a745a5b0476d4e277ae745, State: Running, Role: LEADER
I20260620 07:09:23.987433 7992 consensus_queue.cc:237] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:23.991415 7478 catalog_manager.cc:5495] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7202953660a745a5b0476d4e277ae745 (127.6.53.67). New cstate: current_term: 1 leader_uuid: "7202953660a745a5b0476d4e277ae745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } health_report { overall_health: HEALTHY } } }
Jun 20 07:09:24 dist-test-slave-4n6h krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939361, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:24.036798 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:24.020771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58898 (local address 127.6.53.126:42967)
0620 07:09:24.021460 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:24.021467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:24.021834 (+ 367us) server_negotiation.cc:407] Connection header received
0620 07:09:24.021991 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:24.021997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:24.022074 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:24.022206 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:24.023315 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.024456 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.026197 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.027022 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.027163 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:24.031313 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:24.031318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:24.031322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:24.031348 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:24.034066 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:24.034797 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:24.034803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:24.034808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:24.034930 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:24.035307 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:24.035311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:24.035315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:24.035590 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:24.035740 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:24.036143 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 07:09:24.036361 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":352,"threads_started":1}
I20260620 07:09:24.036877 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:24.020771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42967 (local address 127.0.0.1:58898)
0620 07:09:24.021729 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:24.021746 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:24.021872 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:24.022293 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:24.022300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:24.022941 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:24.023149 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:24.023158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.024596 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.024600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:24.025925 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:24.025943 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.027529 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.027537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:24.027709 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:24.028781 (+ 1072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:24.028804 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:24.031101 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:24.034216 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:24.034228 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:24.034237 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:24.034681 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:24.035042 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:24.035047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:24.035050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:24.035196 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:24.035716 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:24.035724 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:24.035897 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 07:09:24.036057 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 07:09:24.036480 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":154,"thread_start_us":631,"threads_started":1}
I20260620 07:09:24.042197 7478 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:58898:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 07:09:24.043335 7478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:09:24.050308 7617 tablet_service.cc:1467] Processing CreateTablet for tablet f2714ed75935453d8903883f3967e6ae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=871a6408bc854d828cc69f6f565e8ed7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:09:24.050418 7756 tablet_service.cc:1467] Processing CreateTablet for tablet f2714ed75935453d8903883f3967e6ae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=871a6408bc854d828cc69f6f565e8ed7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:09:24.050981 7756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2714ed75935453d8903883f3967e6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:24.050976 7617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2714ed75935453d8903883f3967e6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:24.051105 7895 tablet_service.cc:1467] Processing CreateTablet for tablet f2714ed75935453d8903883f3967e6ae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=871a6408bc854d828cc69f6f565e8ed7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:09:24.051656 7895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2714ed75935453d8903883f3967e6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:24.058573 7985 tablet_bootstrap.cc:492] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e: Bootstrap starting.
I20260620 07:09:24.060016 7985 tablet_bootstrap.cc:654] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:24.060479 7988 tablet_bootstrap.cc:492] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745: Bootstrap starting.
I20260620 07:09:24.061132 7984 tablet_bootstrap.cc:492] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8: Bootstrap starting.
I20260620 07:09:24.062593 7988 tablet_bootstrap.cc:654] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:24.062883 7984 tablet_bootstrap.cc:654] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:24.065550 7988 tablet_bootstrap.cc:492] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745: No bootstrap required, opened a new log
I20260620 07:09:24.065703 7988 ts_tablet_manager.cc:1397] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 07:09:24.066355 7984 tablet_bootstrap.cc:492] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8: No bootstrap required, opened a new log
I20260620 07:09:24.066442 7988 raft_consensus.cc:348] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:24.066672 7988 raft_consensus.cc:374] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:24.066761 7988 raft_consensus.cc:729] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7202953660a745a5b0476d4e277ae745, State: Initialized, Role: FOLLOWER
I20260620 07:09:24.066802 7984 ts_tablet_manager.cc:1397] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:09:24.066996 7988 consensus_queue.cc:260] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:24.067490 7988 ts_tablet_manager.cc:1428] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:09:24.067951 7992 tablet_replica.cc:439] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } } }
I20260620 07:09:24.068289 7992 tablet_replica.cc:442] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:09:24.068742 7984 raft_consensus.cc:348] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:24.069047 7984 raft_consensus.cc:374] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:24.069203 7984 raft_consensus.cc:729] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901ec4eb4d344f35a06e246605a81ba8, State: Initialized, Role: FOLLOWER
I20260620 07:09:24.069525 7984 consensus_queue.cc:260] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:24.070925 7991 tablet_replica.cc:439] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } } }
I20260620 07:09:24.071195 7991 tablet_replica.cc:442] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:09:24.072528 7984 ts_tablet_manager.cc:1428] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260620 07:09:24.073321 7985 tablet_bootstrap.cc:492] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e: No bootstrap required, opened a new log
I20260620 07:09:24.073498 7985 ts_tablet_manager.cc:1397] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20260620 07:09:24.074096 7985 raft_consensus.cc:348] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:24.074446 7985 raft_consensus.cc:374] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:24.074539 7985 raft_consensus.cc:729] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b78ef3d1954520834dd159c7fefd6e, State: Initialized, Role: FOLLOWER
I20260620 07:09:24.074791 7985 consensus_queue.cc:260] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:24.075356 7985 ts_tablet_manager.cc:1428] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:09:24.075294 7990 tablet_replica.cc:439] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } } }
I20260620 07:09:24.075652 7990 tablet_replica.cc:442] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:09:24.241353 7992 raft_consensus.cc:482] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:09:24.241604 7992 raft_consensus.cc:504] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:24.242388 7992 leader_election.cc:288] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64b78ef3d1954520834dd159c7fefd6e (127.6.53.65:45473), 901ec4eb4d344f35a06e246605a81ba8 (127.6.53.66:44793)
I20260620 07:09:24.243151 7638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2714ed75935453d8903883f3967e6ae" candidate_uuid: "7202953660a745a5b0476d4e277ae745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b78ef3d1954520834dd159c7fefd6e" is_pre_election: true
I20260620 07:09:24.243157 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2714ed75935453d8903883f3967e6ae" candidate_uuid: "7202953660a745a5b0476d4e277ae745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901ec4eb4d344f35a06e246605a81ba8" is_pre_election: true
I20260620 07:09:24.243435 7638 raft_consensus.cc:2454] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7202953660a745a5b0476d4e277ae745 in term 0.
I20260620 07:09:24.243445 7780 raft_consensus.cc:2454] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7202953660a745a5b0476d4e277ae745 in term 0.
I20260620 07:09:24.243927 7848 leader_election.cc:302] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7202953660a745a5b0476d4e277ae745, 901ec4eb4d344f35a06e246605a81ba8; no voters:
I20260620 07:09:24.244280 7992 raft_consensus.cc:2781] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:09:24.244382 7992 raft_consensus.cc:482] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:09:24.244481 7992 raft_consensus.cc:3037] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:24.247566 7992 raft_consensus.cc:504] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:24.248142 7992 leader_election.cc:288] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [CANDIDATE]: Term 1 election: Requested vote from peers 64b78ef3d1954520834dd159c7fefd6e (127.6.53.65:45473), 901ec4eb4d344f35a06e246605a81ba8 (127.6.53.66:44793)
I20260620 07:09:24.248646 7780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2714ed75935453d8903883f3967e6ae" candidate_uuid: "7202953660a745a5b0476d4e277ae745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901ec4eb4d344f35a06e246605a81ba8"
I20260620 07:09:24.248698 7638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2714ed75935453d8903883f3967e6ae" candidate_uuid: "7202953660a745a5b0476d4e277ae745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b78ef3d1954520834dd159c7fefd6e"
I20260620 07:09:24.248872 7780 raft_consensus.cc:3037] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:24.248915 7638 raft_consensus.cc:3037] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:24.252269 7780 raft_consensus.cc:2454] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7202953660a745a5b0476d4e277ae745 in term 1.
I20260620 07:09:24.252566 7638 raft_consensus.cc:2454] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7202953660a745a5b0476d4e277ae745 in term 1.
I20260620 07:09:24.252830 7848 leader_election.cc:302] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7202953660a745a5b0476d4e277ae745, 901ec4eb4d344f35a06e246605a81ba8; no voters:
I20260620 07:09:24.253682 7992 raft_consensus.cc:2781] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:24.253934 7992 raft_consensus.cc:686] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [term 1 LEADER]: Becoming Leader. State: Replica: 7202953660a745a5b0476d4e277ae745, State: Running, Role: LEADER
I20260620 07:09:24.254333 7992 consensus_queue.cc:237] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } }
I20260620 07:09:24.255926 8016 tablet_replica.cc:439] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745: TxnStatusTablet state changed. Reason: New leader 7202953660a745a5b0476d4e277ae745. Latest consensus state: current_term: 1 leader_uuid: "7202953660a745a5b0476d4e277ae745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } } }
I20260620 07:09:24.256299 8016 tablet_replica.cc:442] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:09:24.256930 8029 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 07:09:24.257187 8029 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 07:09:24.258060 7480 catalog_manager.cc:5495] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7202953660a745a5b0476d4e277ae745 (127.6.53.67). New cstate: current_term: 1 leader_uuid: "7202953660a745a5b0476d4e277ae745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } health_report { overall_health: HEALTHY } } }
I20260620 07:09:24.314213 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:24.307432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:37175 (local address 127.0.0.1:42400)
0620 07:09:24.307754 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:24.307781 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:24.307947 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:24.308574 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:24.308577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:24.308586 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:09:24.308748 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:24.308754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.310297 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.310302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:24.311338 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:24.311350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.312289 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.312294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:24.312501 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:24.312555 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:09:24.313643 (+ 1088us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:09:24.313664 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 07:09:24.313754 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 07:09:24.313905 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 07:09:24.314273 8032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:24.307395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42400 (local address 127.6.53.67:37175)
0620 07:09:24.308125 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:24.308131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:24.308156 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:09:24.308257 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:24.308262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:24.308359 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:24.308534 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:24.308861 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.310109 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.311477 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.312123 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.312268 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:24.312858 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:24.313519 (+ 661us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:24.313683 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:24.313801 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 07:09:24.313880 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":392,"threads_started":1}
I20260620 07:09:24.318992 7780 raft_consensus.cc:1261] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8 [term 1 FOLLOWER]: Refusing update from remote peer 7202953660a745a5b0476d4e277ae745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:24.319077 7638 raft_consensus.cc:1261] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e [term 1 FOLLOWER]: Refusing update from remote peer 7202953660a745a5b0476d4e277ae745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:24.319702 7992 consensus_queue.cc:1035] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:24.320092 8016 consensus_queue.cc:1035] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:24.323243 7990 tablet_replica.cc:439] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e: TxnStatusTablet state changed. Reason: New leader 7202953660a745a5b0476d4e277ae745. Latest consensus state: current_term: 1 leader_uuid: "7202953660a745a5b0476d4e277ae745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } } }
I20260620 07:09:24.323544 7990 tablet_replica.cc:442] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:09:24.325573 8033 mvcc.cc:205] Tried to move back new op lower bound from 7298823636242903040 to 7298823635988492288. Current Snapshot: MvccSnapshot[applied={T|T < 7298823636242903040}]
I20260620 07:09:24.330773 7992 tablet_replica.cc:439] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7202953660a745a5b0476d4e277ae745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } } }
I20260620 07:09:24.332423 7992 tablet_replica.cc:442] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:09:24.332957 7990 tablet_replica.cc:439] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7202953660a745a5b0476d4e277ae745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } } }
I20260620 07:09:24.333174 7990 tablet_replica.cc:442] T f2714ed75935453d8903883f3967e6ae P 64b78ef3d1954520834dd159c7fefd6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:09:24.336799 7991 tablet_replica.cc:439] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8: TxnStatusTablet state changed. Reason: New leader 7202953660a745a5b0476d4e277ae745. Latest consensus state: current_term: 1 leader_uuid: "7202953660a745a5b0476d4e277ae745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } } }
I20260620 07:09:24.337080 7991 tablet_replica.cc:442] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:09:24.338675 7992 tablet_replica.cc:439] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7202953660a745a5b0476d4e277ae745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } } }
I20260620 07:09:24.338917 7992 tablet_replica.cc:442] T f2714ed75935453d8903883f3967e6ae P 7202953660a745a5b0476d4e277ae745: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:09:24.341506 8035 mvcc.cc:205] Tried to move back new op lower bound from 7298823636242903040 to 7298823635988492288. Current Snapshot: MvccSnapshot[applied={T|T < 7298823636242903040}]
I20260620 07:09:24.343108 7991 tablet_replica.cc:439] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7202953660a745a5b0476d4e277ae745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 } } peers { permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 } } peers { permanent_uuid: "7202953660a745a5b0476d4e277ae745" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37175 } } }
I20260620 07:09:24.343346 7991 tablet_replica.cc:442] T f2714ed75935453d8903883f3967e6ae P 901ec4eb4d344f35a06e246605a81ba8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:09:24.348359 8032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:24.341536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.6.53.67:37175)
0620 07:09:24.342221 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:24.342230 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:24.342905 (+ 675us) server_negotiation.cc:407] Connection header received
0620 07:09:24.343183 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:24.343188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:24.343277 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:24.343339 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:24.344102 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.345031 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.346618 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.347060 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.347210 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:24.347612 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:24.347716 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:24.347827 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:24.347963 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 07:09:24.348057 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527}
I20260620 07:09:24.365913 8032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:24.359533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:59147 (local address 127.6.53.67:37175)
0620 07:09:24.359738 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:24.359745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:24.360410 (+ 665us) server_negotiation.cc:407] Connection header received
0620 07:09:24.360588 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:24.360591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:24.360657 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:24.360728 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:24.361317 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.362030 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.364268 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.364713 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.364805 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:24.365310 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:24.365400 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:24.365464 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:24.365617 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 07:09:24.365676 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:09:24.366291 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:24.359608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:37175 (local address 127.6.53.67:59147)
0620 07:09:24.360310 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:24.360329 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:24.360470 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:24.360767 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:24.360772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:24.360785 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:09:24.361171 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:24.361183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.362157 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.362163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:24.364095 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:24.364110 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.364886 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.364892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:24.365086 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:24.365150 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:09:24.365492 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:09:24.365513 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 07:09:24.365594 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 07:09:24.365794 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":383,"threads_started":1}
I20260620 07:09:24.375877 7638 raft_consensus.cc:1261] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e [term 1 FOLLOWER]: Refusing update from remote peer 7202953660a745a5b0476d4e277ae745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:24.375916 7780 raft_consensus.cc:1261] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8 [term 1 FOLLOWER]: Refusing update from remote peer 7202953660a745a5b0476d4e277ae745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:24.376452 8016 consensus_queue.cc:1035] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "901ec4eb4d344f35a06e246605a81ba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:24.376734 7992 consensus_queue.cc:1035] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64b78ef3d1954520834dd159c7fefd6e" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:24.428486 7480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:09:24.428740 7480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:09:24.431588 7480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51609e677ec04dcc9d19c32001cdb9a2: Sending DeleteTablet for 3 replicas of tablet 98242213c8f24957a87fd423ea2450b4
I20260620 07:09:24.432346 7756 tablet_service.cc:1514] Processing DeleteTablet for tablet 98242213c8f24957a87fd423ea2450b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:24 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:38960
I20260620 07:09:24.432394 7895 tablet_service.cc:1514] Processing DeleteTablet for tablet 98242213c8f24957a87fd423ea2450b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:24 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:42392
I20260620 07:09:24.432942 7617 tablet_service.cc:1514] Processing DeleteTablet for tablet 98242213c8f24957a87fd423ea2450b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:24 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:54034
I20260620 07:09:24.447139 8051 tablet_replica.cc:330] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745: stopping tablet replica
I20260620 07:09:24.447549 8051 raft_consensus.cc:2229] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:09:24.448367 8051 raft_consensus.cc:2258] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:24.449975 8051 ts_tablet_manager.cc:1905] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:24.454612 8050 tablet_replica.cc:330] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8: stopping tablet replica
I20260620 07:09:24.454790 8052 tablet_replica.cc:330] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e: stopping tablet replica
I20260620 07:09:24.454895 8050 raft_consensus.cc:2229] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:09:24.455060 8052 raft_consensus.cc:2229] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:09:24.455236 8050 raft_consensus.cc:2258] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:24.455328 8052 raft_consensus.cc:2258] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:24.456367 8050 ts_tablet_manager.cc:1905] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:24.456697 8052 ts_tablet_manager.cc:1905] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:24.458936 8051 ts_tablet_manager.cc:1918] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:09:24.459125 8051 log.cc:1196] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/98242213c8f24957a87fd423ea2450b4
I20260620 07:09:24.459842 8051 ts_tablet_manager.cc:1939] T 98242213c8f24957a87fd423ea2450b4 P 7202953660a745a5b0476d4e277ae745: Deleting consensus metadata
I20260620 07:09:24.461059 7465 catalog_manager.cc:4841] TS 7202953660a745a5b0476d4e277ae745 (127.6.53.67:37175): tablet 98242213c8f24957a87fd423ea2450b4 (table test-table [id=a8813780ca0c43a8b915b8f24aa27192]) successfully deleted
I20260620 07:09:24.463439 8050 ts_tablet_manager.cc:1918] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:09:24.463652 8050 log.cc:1196] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/98242213c8f24957a87fd423ea2450b4
I20260620 07:09:24.463692 8052 ts_tablet_manager.cc:1918] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:09:24.463888 8052 log.cc:1196] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/98242213c8f24957a87fd423ea2450b4
I20260620 07:09:24.464181 8050 ts_tablet_manager.cc:1939] T 98242213c8f24957a87fd423ea2450b4 P 901ec4eb4d344f35a06e246605a81ba8: Deleting consensus metadata
I20260620 07:09:24.464350 8052 ts_tablet_manager.cc:1939] T 98242213c8f24957a87fd423ea2450b4 P 64b78ef3d1954520834dd159c7fefd6e: Deleting consensus metadata
I20260620 07:09:24.465274 7466 catalog_manager.cc:4841] TS 901ec4eb4d344f35a06e246605a81ba8 (127.6.53.66:44793): tablet 98242213c8f24957a87fd423ea2450b4 (table test-table [id=a8813780ca0c43a8b915b8f24aa27192]) successfully deleted
I20260620 07:09:24.465574 7466 catalog_manager.cc:4841] TS 64b78ef3d1954520834dd159c7fefd6e (127.6.53.65:45473): tablet 98242213c8f24957a87fd423ea2450b4 (table test-table [id=a8813780ca0c43a8b915b8f24aa27192]) successfully deleted
Jun 20 07:09:24 dist-test-slave-4n6h krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939363, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:09:24.473528 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:24.447108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.6.53.65:45473)
0620 07:09:24.447301 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:24.447308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:24.454674 (+ 7366us) server_negotiation.cc:407] Connection header received
0620 07:09:24.455181 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:24.455187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:24.455280 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:24.455375 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:24.457820 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.459045 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.461195 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.461862 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.462001 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:24.467116 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:24.467130 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:24.467151 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:24.467206 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:24.470852 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:24.471525 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:24.471535 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:24.471539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:24.471694 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:24.472033 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:24.472038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:24.472042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:24.472452 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:24.472765 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:24.472907 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 07:09:24.473141 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 07:09:24.474236 7682 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54038
I20260620 07:09:24.490585 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:24.477330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58912 (local address 127.6.53.126:42967)
0620 07:09:24.477525 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:24.477530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:24.477734 (+ 204us) server_negotiation.cc:407] Connection header received
0620 07:09:24.477847 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:24.477852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:24.477945 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:24.478060 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:24.479259 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.480382 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.481523 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.482130 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.482279 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:24.484430 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:24.484436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:24.484440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:24.484473 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:24.487844 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:24.488427 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:24.488433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:24.488437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:24.488587 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:24.488971 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:24.488977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:24.488980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:24.489325 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:24.489559 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:24.489945 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 07:09:24.490177 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 07:09:24.491205 7480 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58912
I20260620 07:09:24.493636 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 7551
I20260620 07:09:24.506865 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:24.494554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54046 (local address 127.6.53.65:45473)
0620 07:09:24.494748 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:24.494756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:24.495058 (+ 302us) server_negotiation.cc:407] Connection header received
0620 07:09:24.495262 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:24.495266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:24.495326 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:24.495428 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:24.496452 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.497168 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.498409 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.498809 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.498944 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:24.501132 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:24.501136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:24.501140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:24.501168 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:24.504018 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:24.504820 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:24.504833 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:24.504837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:24.504961 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:24.505371 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:24.505379 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:24.505382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:24.505649 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:24.505944 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:24.506278 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 07:09:24.506490 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 07:09:24.507499 7682 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54046
W20260620 07:09:24.508091 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 7551: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:09:24.508270 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 7551
I20260620 07:09:24.531427 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 7691
Jun 20 07:09:24 dist-test-slave-4n6h krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939363, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:09:24.547925 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:24.532222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38984 (local address 127.6.53.66:44793)
0620 07:09:24.532695 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:24.532700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:24.532715 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:09:24.532765 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:24.532769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:24.532836 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:24.532922 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:24.534449 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.535165 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.536895 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.537292 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.537387 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:24.542129 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:24.542141 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:24.542156 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:24.542191 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:24.545185 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:24.545918 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:24.545924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:24.545927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:24.546064 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:24.546502 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:24.546507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:24.546510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:24.546849 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:24.547160 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:24.547396 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 07:09:24.547611 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":281,"threads_started":1}
W20260620 07:09:24.548626 7821 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38984
W20260620 07:09:24.549297 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 7691: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:09:24.549494 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 7691
I20260620 07:09:24.581398 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 7830
Jun 20 07:09:24 dist-test-slave-4n6h krb5kdc[7435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939363, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:09:24.602010 8032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:24.582313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.6.53.67:37175)
0620 07:09:24.582505 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:24.582511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:24.582766 (+ 255us) server_negotiation.cc:407] Connection header received
0620 07:09:24.583002 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:24.583008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:24.583084 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:24.583151 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:24.584916 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.586149 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.588021 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:24.588778 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:24.588930 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:24.594446 (+ 5516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:24.594460 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:24.594478 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:24.594522 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:24.598734 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:24.599767 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:24.599776 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:24.599780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:24.599932 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:24.600361 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:24.600366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:24.600370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:24.600797 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:24.601129 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:24.601461 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 07:09:24.601677 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 07:09:24.602929 7960 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42416
W20260620 07:09:24.603767 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 7830: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:09:24.603919 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 7830
I20260620 07:09:24.624727 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 7448
I20260620 07:09:24.626084 7520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:24.738826 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 7448
2026-06-20T07:09:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4836 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:09:24 dist-test-slave-4n6h krb5kdc[8062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:09:24 dist-test-slave-4n6h krb5kdc[8062](info): set up 2 sockets
Jun 20 07:09:24 dist-test-slave-4n6h krb5kdc[8062](info): commencing operation
krb5kdc: starting...
W20260620 07:09:25.826391 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:09:25 dist-test-slave-4n6h krb5kdc[8062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:09:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:09:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:09:26.006318 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:43215
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:43215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:26.330857 8075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:26.331164 8075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:26.331350 8075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:26.339090 8075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:26.339197 8075 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 07:09:26.339269 8075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:26.339373 8075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:26.339465 8075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:26.339587 8075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:26.349891 8075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:43215
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:43215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:26.351639 8075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:26.353148 8075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:26.365669 8080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:26.365690 8081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:26.365738 8083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:26.366504 8075 server_base.cc:1034] running on GCE node
I20260620 07:09:26.367378 8075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:26.368911 8075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:26.370160 8075 hybrid_clock.cc:648] HybridClock initialized: now 1781939366370088 us; error 72 us; skew 500 ppm
I20260620 07:09:26.370620 8075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:26 dist-test-slave-4n6h krb5kdc[8062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939366, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:26.374091 8075 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:09:26.375803 8075 webserver.cc:460] Webserver started at http://127.6.53.126:45325/ using document root <none> and password file <none>
I20260620 07:09:26.376538 8075 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:26.376691 8075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:26.377045 8075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:26.380300 8075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "b7b757f9a3ff4c008ccdb7624c15b4f9"
format_stamp: "Formatted at 2026-06-20 07:09:26 on dist-test-slave-4n6h"
server_key: "7c42a4edd0bfd4857e86f761a748caef"
server_key_iv: "9b3a752e5d39b75a167c56cbd4ea1a7f"
server_key_version: "encryptionkey@0"
I20260620 07:09:26.381268 8075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "b7b757f9a3ff4c008ccdb7624c15b4f9"
format_stamp: "Formatted at 2026-06-20 07:09:26 on dist-test-slave-4n6h"
server_key: "7c42a4edd0bfd4857e86f761a748caef"
server_key_iv: "9b3a752e5d39b75a167c56cbd4ea1a7f"
server_key_version: "encryptionkey@0"
I20260620 07:09:26.386637 8075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:09:26.390087 8090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:26.391465 8075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:26.391652 8075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "b7b757f9a3ff4c008ccdb7624c15b4f9"
format_stamp: "Formatted at 2026-06-20 07:09:26 on dist-test-slave-4n6h"
server_key: "7c42a4edd0bfd4857e86f761a748caef"
server_key_iv: "9b3a752e5d39b75a167c56cbd4ea1a7f"
server_key_version: "encryptionkey@0"
I20260620 07:09:26.391963 8075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:26.433880 8075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:26.437815 8075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:26.438093 8075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:26.459847 8075 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:43215
I20260620 07:09:26.459856 8151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:43215 every 8 connection(s)
I20260620 07:09:26.461194 8075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:26.464512 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 8075
I20260620 07:09:26.464722 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:09:26.464987 8152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:26.465448 6357 external_mini_cluster.cc:1428] Setting key 56688ec7fa95feaf54acdd4b8d62e0c5
I20260620 07:09:26.478972 8152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9: Bootstrap starting.
Jun 20 07:09:26 dist-test-slave-4n6h krb5kdc[8062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:26.482723 8152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:26.483757 8152 log.cc:824] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:26.486394 8152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9: No bootstrap required, opened a new log
I20260620 07:09:26.487980 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:26.467456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.6.53.126:43215)
0620 07:09:26.468362 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:26.468377 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:26.468435 (+ 58us) server_negotiation.cc:407] Connection header received
0620 07:09:26.470088 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:26.470115 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:26.470635 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:26.471078 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:26.472436 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:26.474314 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:26.475506 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:26.476623 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:26.476811 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:26.481108 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:26.481116 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:26.481128 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:26.481169 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:26.484274 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:26.484927 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:26.484941 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:26.484946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:26.485098 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:26.485472 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:26.485481 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:26.485483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:26.485941 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:26.486279 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:26.486678 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 07:09:26.487074 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":390,"threads_started":1}
I20260620 07:09:26.494405 8152 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b757f9a3ff4c008ccdb7624c15b4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 43215 } }
I20260620 07:09:26.494746 8152 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:26.494851 8152 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b757f9a3ff4c008ccdb7624c15b4f9, State: Initialized, Role: FOLLOWER
I20260620 07:09:26.495445 8152 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b757f9a3ff4c008ccdb7624c15b4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 43215 } }
I20260620 07:09:26.495700 8152 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:09:26.495839 8152 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:09:26.496011 8152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:26.498430 8152 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b757f9a3ff4c008ccdb7624c15b4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 43215 } }
I20260620 07:09:26.498874 8152 leader_election.cc:302] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7b757f9a3ff4c008ccdb7624c15b4f9; no voters:
I20260620 07:09:26.499395 8152 leader_election.cc:288] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:09:26.499521 8157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:26.500273 8157 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [term 1 LEADER]: Becoming Leader. State: Replica: b7b757f9a3ff4c008ccdb7624c15b4f9, State: Running, Role: LEADER
I20260620 07:09:26.500746 8157 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b757f9a3ff4c008ccdb7624c15b4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 43215 } }
I20260620 07:09:26.501412 8152 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:26.503638 8158 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7b757f9a3ff4c008ccdb7624c15b4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b757f9a3ff4c008ccdb7624c15b4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 43215 } } }
I20260620 07:09:26.503973 8158 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:26.504312 8159 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7b757f9a3ff4c008ccdb7624c15b4f9. Latest consensus state: current_term: 1 leader_uuid: "b7b757f9a3ff4c008ccdb7624c15b4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b757f9a3ff4c008ccdb7624c15b4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 43215 } } }
I20260620 07:09:26.504628 8159 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:26.506352 8163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:09:26.511487 8163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:09:26.519619 8163 catalog_manager.cc:1347] Generated new cluster ID: c5f2e4cafeb14d62818478ba1e83dba3
I20260620 07:09:26.519721 8163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:26.534896 8163 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:26.536654 8163 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:26.552470 8163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9: Generated new TSK 0
I20260620 07:09:26.553411 8163 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:09:26.608142 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:43215
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:26.938659 8178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:26.938925 8178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:26.939023 8178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:26.945907 8178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:26.946012 8178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:26.946095 8178 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 07:09:26.946147 8178 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:09:26.946214 8178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:09:26.955672 8178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.53.126:43215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:26.956938 8178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:26.958307 8178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:26.971052 8183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:26.971105 8186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:26.971745 8184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:26.973042 8178 server_base.cc:1034] running on GCE node
I20260620 07:09:26.973626 8178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:26.974574 8178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:26.975781 8178 hybrid_clock.cc:648] HybridClock initialized: now 1781939366975737 us; error 29 us; skew 500 ppm
I20260620 07:09:26.976049 8178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:26 dist-test-slave-4n6h krb5kdc[8062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939366, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:26.978998 8178 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:09:26.980391 8178 webserver.cc:460] Webserver started at http://127.6.53.65:42961/ using document root <none> and password file <none>
I20260620 07:09:26.981061 8178 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:26.981197 8178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:26.981570 8178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:26.984395 8178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "fb51a1e996ce4d68b6f3f3c6791d266f"
format_stamp: "Formatted at 2026-06-20 07:09:26 on dist-test-slave-4n6h"
server_key: "35a4276077e102e75f4a44bc2a94f0d1"
server_key_iv: "b09e2a9003fb7eaaca9aa560fc82703f"
server_key_version: "encryptionkey@0"
I20260620 07:09:26.985160 8178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "fb51a1e996ce4d68b6f3f3c6791d266f"
format_stamp: "Formatted at 2026-06-20 07:09:26 on dist-test-slave-4n6h"
server_key: "35a4276077e102e75f4a44bc2a94f0d1"
server_key_iv: "b09e2a9003fb7eaaca9aa560fc82703f"
server_key_version: "encryptionkey@0"
I20260620 07:09:26.989987 8178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260620 07:09:26.993191 8193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:26.994436 8178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:09:26.994599 8178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "fb51a1e996ce4d68b6f3f3c6791d266f"
format_stamp: "Formatted at 2026-06-20 07:09:26 on dist-test-slave-4n6h"
server_key: "35a4276077e102e75f4a44bc2a94f0d1"
server_key_iv: "b09e2a9003fb7eaaca9aa560fc82703f"
server_key_version: "encryptionkey@0"
I20260620 07:09:26.994792 8178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:27.027657 8178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:27.031576 8178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:27.031864 8178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:27.034356 8178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:27.034484 8178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:27.034621 8178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:27.034713 8178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:09:27 dist-test-slave-4n6h krb5kdc[8062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939366, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:27.057482 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:27.035098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:49317 (local address 127.6.53.126:43215)
0620 07:09:27.035732 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:27.035738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:27.037200 (+ 1462us) server_negotiation.cc:407] Connection header received
0620 07:09:27.039934 (+ 2734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:27.039940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:27.040026 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:27.040154 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:27.042325 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:27.043582 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:27.045119 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:27.045921 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:27.046072 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:27.050747 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:27.050768 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:27.050775 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:27.050827 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:27.054641 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:27.055319 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:27.055326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:27.055330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:27.055476 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:27.055898 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:27.055903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:27.055907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:27.056177 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:27.056324 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:27.056921 (+ 597us) server_negotiation.cc:299] Negotiation successful
0620 07:09:27.057124 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":403,"threads_started":1}
I20260620 07:09:27.058789 8203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:27.035817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:43215 (local address 127.6.53.65:49317)
0620 07:09:27.036731 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:27.036781 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:27.039640 (+ 2859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:27.040429 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:27.040443 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:27.041210 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:27.042028 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:27.042051 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:27.043751 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:27.043755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:27.044980 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:27.044991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:27.046174 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:27.046187 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:27.046500 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:27.047312 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:27.047347 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:27.050466 (+ 3119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:27.054768 (+ 4302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:27.054778 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:27.054792 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:27.055191 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:27.055583 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:27.055590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:27.055594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:27.055780 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:27.056296 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:27.056303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:27.056607 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 07:09:27.056853 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 07:09:27.057321 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":387,"threads_started":1}
I20260620 07:09:27.078915 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:27.065291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:44723 (local address 127.6.53.126:43215)
0620 07:09:27.065478 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:27.065484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:27.066190 (+ 706us) server_negotiation.cc:407] Connection header received
0620 07:09:27.066453 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:27.066459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:27.066553 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:27.066650 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:27.068098 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:27.069136 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:27.071078 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:27.071492 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:27.071614 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:27.073221 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:27.073227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:27.073234 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:27.073271 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:27.076428 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:27.077049 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:27.077056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:27.077060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:27.077186 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:27.077536 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:27.077543 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:27.077546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:27.077808 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:27.077952 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:27.078365 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 07:09:27.078567 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:09:27.079052 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:27.065287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:43215 (local address 127.6.53.65:44723)
0620 07:09:27.066068 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:27.066093 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:27.066286 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:27.066791 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:27.066798 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:27.067460 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:27.067910 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:27.067929 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:27.069276 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:27.069280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:27.070941 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:27.070952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:27.071672 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:27.071678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:27.071838 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:27.072409 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:27.072425 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:27.073051 (+ 626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:27.076547 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:27.076554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:27.076558 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:27.076934 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:27.077294 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:27.077299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:27.077303 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:27.077422 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:27.077918 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:27.077923 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:27.078088 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 07:09:27.078242 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 07:09:27.078576 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":425,"threads_started":1}
I20260620 07:09:27.092397 8178 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:38753
I20260620 07:09:27.092478 8313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:38753 every 8 connection(s)
I20260620 07:09:27.093623 8178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:09:27.095675 8314 heartbeater.cc:347] Connected to a master server at 127.6.53.126:43215
I20260620 07:09:27.095867 8314 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:27.096416 8314 heartbeater.cc:511] Master 127.6.53.126:43215 requested a full tablet report, sending...
I20260620 07:09:27.096412 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 8178
I20260620 07:09:27.096644 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:09:27.097317 6357 external_mini_cluster.cc:1428] Setting key 1f8e0d4a5dcb28cd75606e9600bedafb
I20260620 07:09:27.098835 8107 ts_manager.cc:194] Registered new tserver with Master: fb51a1e996ce4d68b6f3f3c6791d266f (127.6.53.65:38753)
I20260620 07:09:27.100770 8107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:49317
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:09:27.135061 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:43215
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:27.441154 8318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:27.441474 8318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:27.441656 8318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:27.448827 8318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:27.448983 8318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:27.449141 8318 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 07:09:27.449229 8318 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:09:27.449373 8318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:09:27.459335 8318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.53.126:43215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:27.460867 8318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:27.462522 8318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:27.476791 8324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:27.476806 8323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:27.477465 8326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:27.477530 8318 server_base.cc:1034] running on GCE node
I20260620 07:09:27.478175 8318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:27.479166 8318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:27.480386 8318 hybrid_clock.cc:648] HybridClock initialized: now 1781939367480335 us; error 58 us; skew 500 ppm
I20260620 07:09:27.480684 8318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:27 dist-test-slave-4n6h krb5kdc[8062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939367, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:27.483675 8318 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:09:27.485062 8318 webserver.cc:460] Webserver started at http://127.6.53.66:41185/ using document root <none> and password file <none>
I20260620 07:09:27.485698 8318 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:27.485848 8318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:27.486147 8318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:27.488916 8318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "9dc24236063c4c7aa912060d76892563"
format_stamp: "Formatted at 2026-06-20 07:09:27 on dist-test-slave-4n6h"
server_key: "8dffba389757653e81351f315746672a"
server_key_iv: "91a3a2f6512e3ffc5cddff16d30f9432"
server_key_version: "encryptionkey@0"
I20260620 07:09:27.489624 8318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "9dc24236063c4c7aa912060d76892563"
format_stamp: "Formatted at 2026-06-20 07:09:27 on dist-test-slave-4n6h"
server_key: "8dffba389757653e81351f315746672a"
server_key_iv: "91a3a2f6512e3ffc5cddff16d30f9432"
server_key_version: "encryptionkey@0"
I20260620 07:09:27.494511 8318 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260620 07:09:27.497803 8333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:27.499428 8318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:27.499567 8318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "9dc24236063c4c7aa912060d76892563"
format_stamp: "Formatted at 2026-06-20 07:09:27 on dist-test-slave-4n6h"
server_key: "8dffba389757653e81351f315746672a"
server_key_iv: "91a3a2f6512e3ffc5cddff16d30f9432"
server_key_version: "encryptionkey@0"
I20260620 07:09:27.499717 8318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:27.532512 8318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:27.536774 8318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:27.537003 8318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:27.539613 8318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:27.539815 8318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 07:09:27.539953 8318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:27.540050 8318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:09:27 dist-test-slave-4n6h krb5kdc[8062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939367, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:27.567529 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:27.540054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:55209 (local address 127.6.53.126:43215)
0620 07:09:27.540295 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:27.540301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:27.541670 (+ 1369us) server_negotiation.cc:407] Connection header received
0620 07:09:27.543897 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:27.543903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:27.544028 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:27.544180 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:27.546444 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:27.547513 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:27.549906 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:27.550386 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:27.550520 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:27.557877 (+ 7357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:27.557886 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:27.557895 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:27.557965 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:27.562940 (+ 4975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:27.564142 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:27.564150 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:27.564156 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:27.564356 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:27.564929 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:27.564936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:27.564942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:27.565406 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:27.565608 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:27.566748 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0620 07:09:27.567049 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:09:27.570642 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:27.540628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:43215 (local address 127.6.53.66:55209)
0620 07:09:27.541481 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:27.541536 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:27.543576 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:27.544592 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:27.544603 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:27.545317 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:27.546221 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:27.546272 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:27.547690 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:27.547696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:27.549753 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:27.549764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:27.550644 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:27.550657 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:27.551092 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:27.552254 (+ 1162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:27.552292 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:27.556570 (+ 4278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:27.563132 (+ 6562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:27.563144 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:27.563164 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:27.563972 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:27.564491 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:27.564499 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:27.564505 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:27.564782 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:27.565696 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:27.565708 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:27.566178 (+ 470us) client_negotiation.cc:769] Sending connection context
0620 07:09:27.568030 (+ 1852us) client_negotiation.cc:240] Negotiation successful
0620 07:09:27.568695 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":373,"threads_started":1}
I20260620 07:09:27.602105 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:27.582540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:43215 (local address 127.6.53.66:33171)
0620 07:09:27.583344 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:27.583370 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:27.583563 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:27.584154 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:27.584161 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:27.584939 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:27.585538 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:27.585562 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:27.587430 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:27.587437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:27.589661 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:27.589684 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:27.591171 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:27.591180 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:27.591439 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:27.592174 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:27.592204 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:27.593408 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:27.597998 (+ 4590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:27.598003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:27.598006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:27.598711 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:27.599223 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:27.599231 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:27.599234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:27.599429 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:27.600373 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:27.600380 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:27.600663 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 07:09:27.600853 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 07:09:27.601376 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":406,"threads_started":1}
I20260620 07:09:27.602218 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:27.582477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:33171 (local address 127.6.53.126:43215)
0620 07:09:27.582678 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:27.582687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:27.583448 (+ 761us) server_negotiation.cc:407] Connection header received
0620 07:09:27.583730 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:27.583738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:27.583829 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:27.583903 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:27.585758 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:27.587211 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:27.589858 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:27.590683 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:27.590794 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:27.593659 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:27.593666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:27.593671 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:27.593707 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:27.597836 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:27.598884 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:27.598892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:27.598899 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:27.599079 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:27.599585 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:27.599593 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:27.599599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:27.600100 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:27.600275 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:27.601434 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0620 07:09:27.601726 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:09:27.607491 8318 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:42645
I20260620 07:09:27.607637 8452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:42645 every 8 connection(s)
I20260620 07:09:27.608721 8318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:09:27.610720 8453 heartbeater.cc:347] Connected to a master server at 127.6.53.126:43215
I20260620 07:09:27.610952 8453 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:27.611598 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 8318
I20260620 07:09:27.611743 8453 heartbeater.cc:511] Master 127.6.53.126:43215 requested a full tablet report, sending...
I20260620 07:09:27.611784 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:09:27.612318 6357 external_mini_cluster.cc:1428] Setting key a7d59012bd7d4f14ab1f351b7d6c4d00
I20260620 07:09:27.614290 8107 ts_manager.cc:194] Registered new tserver with Master: 9dc24236063c4c7aa912060d76892563 (127.6.53.66:42645)
I20260620 07:09:27.615607 8107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:55209
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:09:27.659559 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:43215
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:28.013517 8457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:28.013842 8457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:28.014012 8457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:28.021291 8457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:28.021447 8457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:28.021587 8457 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 07:09:28.021677 8457 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:09:28.021813 8457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:09:28.031607 8457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.53.126:43215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:28.032866 8457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:28.034119 8457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:28.047660 8463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:28.047709 8465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:28.047758 8462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:28.051635 8457 server_base.cc:1034] running on GCE node
I20260620 07:09:28.052253 8457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:28.053146 8457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:28.054370 8457 hybrid_clock.cc:648] HybridClock initialized: now 1781939368054329 us; error 58 us; skew 500 ppm
I20260620 07:09:28.054673 8457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:28 dist-test-slave-4n6h krb5kdc[8062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939368, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:28.061003 8457 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:09:28.062564 8457 webserver.cc:460] Webserver started at http://127.6.53.67:36543/ using document root <none> and password file <none>
I20260620 07:09:28.063319 8457 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:28.063442 8457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:28.063769 8457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:28.066642 8457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "2c65855a594b4570b8139916901d4181"
format_stamp: "Formatted at 2026-06-20 07:09:28 on dist-test-slave-4n6h"
server_key: "6ad67bfce9562a2ab6b1fd36073db403"
server_key_iv: "7b5a78da204b5cd874747d0cc47a0be5"
server_key_version: "encryptionkey@0"
I20260620 07:09:28.067440 8457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "2c65855a594b4570b8139916901d4181"
format_stamp: "Formatted at 2026-06-20 07:09:28 on dist-test-slave-4n6h"
server_key: "6ad67bfce9562a2ab6b1fd36073db403"
server_key_iv: "7b5a78da204b5cd874747d0cc47a0be5"
server_key_version: "encryptionkey@0"
I20260620 07:09:28.072546 8457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:09:28.075894 8472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:28.077302 8457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:09:28.077456 8457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "2c65855a594b4570b8139916901d4181"
format_stamp: "Formatted at 2026-06-20 07:09:28 on dist-test-slave-4n6h"
server_key: "6ad67bfce9562a2ab6b1fd36073db403"
server_key_iv: "7b5a78da204b5cd874747d0cc47a0be5"
server_key_version: "encryptionkey@0"
I20260620 07:09:28.077637 8457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:28.101217 8457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:28.103655 8314 heartbeater.cc:503] Master 127.6.53.126:43215 was elected leader, sending a full tablet report...
I20260620 07:09:28.106200 8457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:28.106580 8457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:28.122213 8457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:28.122383 8457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 07:09:28.122563 8457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:28.122666 8457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:09:28 dist-test-slave-4n6h krb5kdc[8062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939368, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:28.137670 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.110604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:48011 (local address 127.6.53.126:43215)
0620 07:09:28.111204 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:28.111210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:28.114076 (+ 2866us) server_negotiation.cc:407] Connection header received
0620 07:09:28.116775 (+ 2699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:28.116785 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:28.116897 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:28.117048 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:28.120082 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.121626 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.124342 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.124763 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.124866 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.130175 (+ 5309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:28.130182 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:28.130187 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:28.130233 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:28.133457 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:28.134618 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:28.134624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:28.134628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:28.134797 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:28.135382 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:28.135390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:28.135395 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:28.135823 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:28.135990 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:28.136745 (+ 755us) server_negotiation.cc:299] Negotiation successful
0620 07:09:28.137064 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":325,"threads_started":1}
I20260620 07:09:28.138119 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.112842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:43215 (local address 127.6.53.67:48011)
0620 07:09:28.113918 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:28.113963 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:28.116436 (+ 2473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:28.117602 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:28.117618 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:28.118790 (+ 1172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:28.119811 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.119842 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.121820 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.121827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.124176 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.124191 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.125007 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.125017 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.125367 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.126123 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:28.126164 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:28.129852 (+ 3688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:28.133627 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:28.133641 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:28.133658 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:28.134449 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:28.134901 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:28.134906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:28.134911 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:28.135206 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:28.135985 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:28.135997 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:28.136332 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 07:09:28.136552 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 07:09:28.136926 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":451,"threads_started":1}
I20260620 07:09:28.161410 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.145469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:37989 (local address 127.6.53.126:43215)
0620 07:09:28.145657 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:28.145665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:28.146376 (+ 711us) server_negotiation.cc:407] Connection header received
0620 07:09:28.146649 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:28.146655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:28.146729 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:28.146832 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:28.148111 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.149565 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.151188 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.151796 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.151965 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.154842 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:28.154849 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:28.154853 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:28.154890 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:28.158163 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:28.159075 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:28.159080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:28.159084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:28.159227 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:28.159741 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:28.159748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:28.159751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:28.160137 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:28.160294 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:28.160865 (+ 571us) server_negotiation.cc:299] Negotiation successful
0620 07:09:28.161085 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:09:28.161499 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.145445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:43215 (local address 127.6.53.67:37989)
0620 07:09:28.146238 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:28.146291 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:28.146446 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:28.146855 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:28.146859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:28.147549 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:28.147931 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.147940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.149711 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.149714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.151033 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.151048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.152071 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.152078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.152442 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.153175 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:28.153205 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:28.154637 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:28.158311 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:28.158316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:28.158319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:28.158931 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:28.159375 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:28.159380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:28.159384 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:28.159585 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:28.160355 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:28.160358 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:28.160570 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 07:09:28.160707 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 07:09:28.161040 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":387,"threads_started":1}
I20260620 07:09:28.171149 8457 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:38981
I20260620 07:09:28.171204 8592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:38981 every 8 connection(s)
I20260620 07:09:28.172291 8457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:09:28.174445 8593 heartbeater.cc:347] Connected to a master server at 127.6.53.126:43215
I20260620 07:09:28.174681 8593 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:28.175298 8593 heartbeater.cc:511] Master 127.6.53.126:43215 requested a full tablet report, sending...
I20260620 07:09:28.176661 8107 ts_manager.cc:194] Registered new tserver with Master: 2c65855a594b4570b8139916901d4181 (127.6.53.67:38981)
I20260620 07:09:28.177624 8107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:48011
I20260620 07:09:28.178983 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 8457
I20260620 07:09:28.179147 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:09:28.179695 6357 external_mini_cluster.cc:1428] Setting key 40fc51d6c37c00009c9bd71c2d179e29
I20260620 07:09:28.182574 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 07:09:28 dist-test-slave-4n6h krb5kdc[8062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:09:28 dist-test-slave-4n6h krb5kdc[8062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:28.228458 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.214812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.6.53.126:43215)
0620 07:09:28.215041 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:28.215047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:28.215258 (+ 211us) server_negotiation.cc:407] Connection header received
0620 07:09:28.215480 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:28.215484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:28.215550 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:28.215667 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:28.217091 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.217837 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.219154 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.219606 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.219739 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.223430 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:28.223435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:28.223438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:28.223469 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:28.225863 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:28.226538 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:28.226544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:28.226547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:28.226679 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:28.227058 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:28.227063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:28.227068 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:28.227334 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:28.227601 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:28.227895 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 07:09:28.228105 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:09:28.232287 8107 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:09:28.239148 8107 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:09:28.262223 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.250385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:38981 (local address 127.0.0.1:42754)
0620 07:09:28.251893 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:28.251943 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:28.252150 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:28.253778 (+ 1628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:28.253785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:28.253815 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:28.254280 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.254301 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.256465 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.256472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.259238 (+ 2766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.259253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.261023 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.261058 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.261272 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.261329 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 07:09:28.261494 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 07:09:28.261701 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1193,"mutex_wait_us":107,"thread_start_us":1006,"threads_started":1}
I20260620 07:09:28.262425 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.251790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49238 (local address 127.6.53.66:42645)
0620 07:09:28.253219 (+ 1429us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:28.253232 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:28.254599 (+ 1367us) server_negotiation.cc:407] Connection header received
0620 07:09:28.254919 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:28.254926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:28.255136 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:28.255283 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:28.256042 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.257276 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.259746 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.261081 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.261202 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.261394 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:28.261963 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 07:09:28.262136 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1247,"thread_start_us":1146,"threads_started":1}
I20260620 07:09:28.262909 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.250816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.6.53.67:38981)
0620 07:09:28.253062 (+ 2246us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:28.253074 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:28.253115 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:09:28.253235 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:28.253246 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:28.253541 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:28.253750 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:28.254536 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.256319 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.259421 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.260719 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.260869 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.262160 (+ 1291us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:28.262249 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 07:09:28.262478 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2032,"thread_start_us":1816,"threads_started":1}
I20260620 07:09:28.263037 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.251408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:42645 (local address 127.0.0.1:49238)
0620 07:09:28.254498 (+ 3090us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:28.254515 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:28.254716 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:28.255497 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:28.255501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:28.255523 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:28.255848 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.255858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.257451 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.257456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.259584 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.259598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.261533 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.261539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.261781 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.261810 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 07:09:28.262595 (+ 785us) client_negotiation.cc:240] Negotiation successful
0620 07:09:28.262722 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2963,"mutex_wait_us":127,"thread_start_us":2008,"threads_started":1}
I20260620 07:09:28.263197 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.250834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.6.53.65:38753)
0620 07:09:28.251397 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:28.251408 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:28.254113 (+ 2705us) server_negotiation.cc:407] Connection header received
0620 07:09:28.254229 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:28.254236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:28.254486 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:28.254667 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:28.255465 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.256744 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.259486 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.260937 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.261067 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.261287 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:28.262728 (+ 1441us) server_negotiation.cc:299] Negotiation successful
0620 07:09:28.262895 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":266,"threads_started":1}
I20260620 07:09:28.263520 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.250555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38753 (local address 127.0.0.1:54110)
0620 07:09:28.253751 (+ 3196us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:28.253773 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:28.253945 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:28.254913 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:28.254919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:28.254941 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:28.255307 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.255317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.256883 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.256888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.259354 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.259367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.261450 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.261459 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.261690 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.261774 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 07:09:28.261861 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 07:09:28.261989 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3020,"mutex_wait_us":94,"thread_start_us":3637,"threads_started":1}
I20260620 07:09:28.266426 8527 tablet_service.cc:1467] Processing CreateTablet for tablet 13395f9492d04b238bd55f36f9207e3f (DEFAULT_TABLE table=test-table [id=aaa3bd6321694612a0451c4a88f8bca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:28.266909 8244 tablet_service.cc:1467] Processing CreateTablet for tablet 13395f9492d04b238bd55f36f9207e3f (DEFAULT_TABLE table=test-table [id=aaa3bd6321694612a0451c4a88f8bca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:28.266909 8383 tablet_service.cc:1467] Processing CreateTablet for tablet 13395f9492d04b238bd55f36f9207e3f (DEFAULT_TABLE table=test-table [id=aaa3bd6321694612a0451c4a88f8bca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:28.267685 8527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13395f9492d04b238bd55f36f9207e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:28.268175 8244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13395f9492d04b238bd55f36f9207e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:28.268190 8383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13395f9492d04b238bd55f36f9207e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:28.279166 8612 tablet_bootstrap.cc:492] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181: Bootstrap starting.
I20260620 07:09:28.282521 8613 tablet_bootstrap.cc:492] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563: Bootstrap starting.
I20260620 07:09:28.283355 8612 tablet_bootstrap.cc:654] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:28.284725 8612 log.cc:824] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:28.285866 8614 tablet_bootstrap.cc:492] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f: Bootstrap starting.
I20260620 07:09:28.286160 8613 tablet_bootstrap.cc:654] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:28.287400 8613 log.cc:824] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:28.289331 8614 tablet_bootstrap.cc:654] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:28.289927 8612 tablet_bootstrap.cc:492] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181: No bootstrap required, opened a new log
I20260620 07:09:28.290086 8613 tablet_bootstrap.cc:492] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563: No bootstrap required, opened a new log
I20260620 07:09:28.290198 8612 ts_tablet_manager.cc:1397] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260620 07:09:28.290372 8613 ts_tablet_manager.cc:1397] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260620 07:09:28.290539 8614 log.cc:824] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:28.293561 8614 tablet_bootstrap.cc:492] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f: No bootstrap required, opened a new log
I20260620 07:09:28.293816 8614 ts_tablet_manager.cc:1397] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 07:09:28.298151 8613 raft_consensus.cc:348] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } }
I20260620 07:09:28.298624 8613 raft_consensus.cc:374] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:28.298735 8613 raft_consensus.cc:729] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc24236063c4c7aa912060d76892563, State: Initialized, Role: FOLLOWER
I20260620 07:09:28.299382 8613 consensus_queue.cc:260] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } }
I20260620 07:09:28.300384 8453 heartbeater.cc:503] Master 127.6.53.126:43215 was elected leader, sending a full tablet report...
I20260620 07:09:28.300556 8612 raft_consensus.cc:348] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } }
I20260620 07:09:28.300956 8612 raft_consensus.cc:374] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:28.301047 8612 raft_consensus.cc:729] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c65855a594b4570b8139916901d4181, State: Initialized, Role: FOLLOWER
I20260620 07:09:28.301630 8612 consensus_queue.cc:260] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } }
I20260620 07:09:28.302402 8613 ts_tablet_manager.cc:1428] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260620 07:09:28.303498 8593 heartbeater.cc:503] Master 127.6.53.126:43215 was elected leader, sending a full tablet report...
I20260620 07:09:28.304070 8612 ts_tablet_manager.cc:1428] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260620 07:09:28.304719 8614 raft_consensus.cc:348] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } }
I20260620 07:09:28.305169 8614 raft_consensus.cc:374] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:28.305285 8614 raft_consensus.cc:729] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb51a1e996ce4d68b6f3f3c6791d266f, State: Initialized, Role: FOLLOWER
I20260620 07:09:28.305859 8614 consensus_queue.cc:260] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } }
I20260620 07:09:28.307744 8614 ts_tablet_manager.cc:1428] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260620 07:09:28.321992 8619 raft_consensus.cc:482] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:09:28.322356 8619 raft_consensus.cc:504] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } }
I20260620 07:09:28.323935 8619 leader_election.cc:288] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb51a1e996ce4d68b6f3f3c6791d266f (127.6.53.65:38753), 9dc24236063c4c7aa912060d76892563 (127.6.53.66:42645)
I20260620 07:09:28.330404 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.324166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38753 (local address 127.6.53.67:46253)
0620 07:09:28.324504 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:28.324529 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:28.324667 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:28.325116 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:28.325119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:28.325142 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:28.325322 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.325330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.326874 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.326880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.328412 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.328423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.329625 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.329633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.329865 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.329895 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:09:28.329964 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 07:09:28.330056 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":79}
I20260620 07:09:28.330439 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.324302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:46253 (local address 127.6.53.65:38753)
0620 07:09:28.324529 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:28.324535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:28.324609 (+ 74us) server_negotiation.cc:407] Connection header received
0620 07:09:28.324892 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:28.324899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:28.324991 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:28.325093 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:28.325794 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.326751 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.328570 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.329469 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.329573 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.329643 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:28.330016 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 07:09:28.330157 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 07:09:28.330587 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.324425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:54377 (local address 127.6.53.66:42645)
0620 07:09:28.324620 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:28.324628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:28.325277 (+ 649us) server_negotiation.cc:407] Connection header received
0620 07:09:28.325460 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:28.325466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:28.325527 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:28.325621 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:28.326180 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.327074 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.328818 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.329590 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.329721 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.329826 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:28.330163 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 07:09:28.330341 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:09:28.330682 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.324344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:42645 (local address 127.6.53.67:54377)
0620 07:09:28.325188 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:28.325207 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:28.325330 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:28.325633 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:28.325638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:28.325661 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:28.326000 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.326010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.327208 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.327212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.328668 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.328682 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.329820 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.329824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.330000 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.330034 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 07:09:28.330121 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 07:09:28.330266 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":133,"thread_start_us":397,"threads_started":1}
I20260620 07:09:28.331517 8269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13395f9492d04b238bd55f36f9207e3f" candidate_uuid: "2c65855a594b4570b8139916901d4181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" is_pre_election: true
I20260620 07:09:28.331725 8407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13395f9492d04b238bd55f36f9207e3f" candidate_uuid: "2c65855a594b4570b8139916901d4181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc24236063c4c7aa912060d76892563" is_pre_election: true
I20260620 07:09:28.332110 8269 raft_consensus.cc:2454] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c65855a594b4570b8139916901d4181 in term 0.
I20260620 07:09:28.332196 8407 raft_consensus.cc:2454] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c65855a594b4570b8139916901d4181 in term 0.
I20260620 07:09:28.332841 8475 leader_election.cc:302] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c65855a594b4570b8139916901d4181, fb51a1e996ce4d68b6f3f3c6791d266f; no voters:
I20260620 07:09:28.333325 8619 raft_consensus.cc:2781] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:09:28.333482 8619 raft_consensus.cc:482] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:09:28.333603 8619 raft_consensus.cc:3037] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:28.335861 8619 raft_consensus.cc:504] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } }
I20260620 07:09:28.336444 8619 leader_election.cc:288] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [CANDIDATE]: Term 1 election: Requested vote from peers fb51a1e996ce4d68b6f3f3c6791d266f (127.6.53.65:38753), 9dc24236063c4c7aa912060d76892563 (127.6.53.66:42645)
I20260620 07:09:28.336937 8407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13395f9492d04b238bd55f36f9207e3f" candidate_uuid: "2c65855a594b4570b8139916901d4181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc24236063c4c7aa912060d76892563"
I20260620 07:09:28.336931 8269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13395f9492d04b238bd55f36f9207e3f" candidate_uuid: "2c65855a594b4570b8139916901d4181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f"
I20260620 07:09:28.337164 8407 raft_consensus.cc:3037] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:28.337157 8269 raft_consensus.cc:3037] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:28.339636 8407 raft_consensus.cc:2454] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c65855a594b4570b8139916901d4181 in term 1.
I20260620 07:09:28.339701 8269 raft_consensus.cc:2454] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c65855a594b4570b8139916901d4181 in term 1.
I20260620 07:09:28.340065 8476 leader_election.cc:302] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c65855a594b4570b8139916901d4181, 9dc24236063c4c7aa912060d76892563; no voters:
I20260620 07:09:28.340458 8619 raft_consensus.cc:2781] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:28.340871 8619 raft_consensus.cc:686] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [term 1 LEADER]: Becoming Leader. State: Replica: 2c65855a594b4570b8139916901d4181, State: Running, Role: LEADER
I20260620 07:09:28.341347 8619 consensus_queue.cc:237] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } }
W20260620 07:09:28.345899 8315 tablet.cc:2369] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:09:28.346369 8106 catalog_manager.cc:5495] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c65855a594b4570b8139916901d4181 (127.6.53.67). New cstate: current_term: 1 leader_uuid: "2c65855a594b4570b8139916901d4181" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } health_report { overall_health: UNKNOWN } } }
W20260620 07:09:28.360738 8454 tablet.cc:2369] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 07:09:28 dist-test-slave-4n6h krb5kdc[8062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939366, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:28.394527 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.380797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.6.53.126:43215)
0620 07:09:28.380962 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:28.380967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:28.381529 (+ 562us) server_negotiation.cc:407] Connection header received
0620 07:09:28.381781 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:28.381784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:28.381872 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:28.381979 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:28.383200 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.383943 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.385159 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.385608 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.385743 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.389002 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:28.389007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:28.389010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:28.389041 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:28.391887 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:28.392552 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:28.392559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:28.392563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:28.392680 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:28.393044 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:28.393051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:28.393057 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:28.393300 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:28.393452 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:28.393833 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 07:09:28.394155 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":58}
I20260620 07:09:28.394534 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.380841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:43215 (local address 127.0.0.1:48874)
0620 07:09:28.381422 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:28.381438 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:28.381649 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:28.381997 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:28.382001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:28.382776 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:28.383066 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.383075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.384064 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.384072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.385016 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.385027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.385783 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.385790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.385954 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.386606 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:28.386622 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:28.388775 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:28.392027 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:28.392040 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:28.392048 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:28.392444 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:28.392786 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:28.392790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:28.392794 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:28.392942 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:28.393428 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:28.393436 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:28.393592 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 07:09:28.393754 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 07:09:28.394035 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":354,"threads_started":1}
I20260620 07:09:28.399761 8106 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:48874:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 07:09:28.400830 8106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:09:28.407620 8244 tablet_service.cc:1467] Processing CreateTablet for tablet 46cce709dfff46b898b3dbc0328d5dc0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1aba3a228a324d0eadcd5da742cd1341]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:09:28.407757 8383 tablet_service.cc:1467] Processing CreateTablet for tablet 46cce709dfff46b898b3dbc0328d5dc0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1aba3a228a324d0eadcd5da742cd1341]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:09:28.408023 8527 tablet_service.cc:1467] Processing CreateTablet for tablet 46cce709dfff46b898b3dbc0328d5dc0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1aba3a228a324d0eadcd5da742cd1341]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:09:28.408604 8383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cce709dfff46b898b3dbc0328d5dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:28.408604 8244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cce709dfff46b898b3dbc0328d5dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:28.408865 8527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cce709dfff46b898b3dbc0328d5dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:28.414304 8612 tablet_bootstrap.cc:492] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181: Bootstrap starting.
I20260620 07:09:28.415681 8613 tablet_bootstrap.cc:492] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563: Bootstrap starting.
I20260620 07:09:28.415891 8612 tablet_bootstrap.cc:654] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:28.417299 8613 tablet_bootstrap.cc:654] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:28.417981 8612 tablet_bootstrap.cc:492] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181: No bootstrap required, opened a new log
I20260620 07:09:28.418131 8612 ts_tablet_manager.cc:1397] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260620 07:09:28.418344 8614 tablet_bootstrap.cc:492] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f: Bootstrap starting.
I20260620 07:09:28.418941 8612 raft_consensus.cc:348] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.419158 8612 raft_consensus.cc:374] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:28.419269 8612 raft_consensus.cc:729] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c65855a594b4570b8139916901d4181, State: Initialized, Role: FOLLOWER
I20260620 07:09:28.419524 8612 consensus_queue.cc:260] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.419766 8614 tablet_bootstrap.cc:654] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:28.419904 8619 tablet_replica.cc:439] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } }
I20260620 07:09:28.420102 8612 ts_tablet_manager.cc:1428] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 07:09:28.420204 8619 tablet_replica.cc:442] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:09:28.421759 8613 tablet_bootstrap.cc:492] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563: No bootstrap required, opened a new log
I20260620 07:09:28.421923 8613 ts_tablet_manager.cc:1397] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260620 07:09:28.421973 8614 tablet_bootstrap.cc:492] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f: No bootstrap required, opened a new log
I20260620 07:09:28.422113 8614 ts_tablet_manager.cc:1397] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 07:09:28.422735 8613 raft_consensus.cc:348] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.422979 8613 raft_consensus.cc:374] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:28.422880 8614 raft_consensus.cc:348] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.423079 8613 raft_consensus.cc:729] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc24236063c4c7aa912060d76892563, State: Initialized, Role: FOLLOWER
I20260620 07:09:28.423115 8614 raft_consensus.cc:374] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:28.423206 8614 raft_consensus.cc:729] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb51a1e996ce4d68b6f3f3c6791d266f, State: Initialized, Role: FOLLOWER
I20260620 07:09:28.423345 8613 consensus_queue.cc:260] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
W20260620 07:09:28.423950 8594 tablet.cc:2369] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:09:28.423820 8618 tablet_replica.cc:439] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } }
I20260620 07:09:28.424221 8618 tablet_replica.cc:442] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:09:28.423518 8614 consensus_queue.cc:260] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.424767 8620 tablet_replica.cc:439] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } }
I20260620 07:09:28.425112 8613 ts_tablet_manager.cc:1428] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 07:09:28.425139 8620 tablet_replica.cc:442] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:09:28.425585 8614 ts_tablet_manager.cc:1428] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 07:09:28.463841 8620 raft_consensus.cc:482] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:09:28.464090 8620 raft_consensus.cc:504] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.465842 8620 leader_election.cc:288] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dc24236063c4c7aa912060d76892563 (127.6.53.66:42645), 2c65855a594b4570b8139916901d4181 (127.6.53.67:38981)
I20260620 07:09:28.473292 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.466497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:49567 (local address 127.6.53.67:38981)
0620 07:09:28.466923 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:28.466931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:28.466955 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:09:28.467046 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:28.467054 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:28.467155 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:28.467278 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:28.467905 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.468993 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.471171 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.472323 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.472460 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.472561 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:28.472899 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 07:09:28.473052 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260620 07:09:28.473902 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.466103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:38981 (local address 127.6.53.65:49567)
0620 07:09:28.466664 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:28.466683 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:28.466808 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:28.467295 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:28.467306 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:28.467332 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:28.467722 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.467733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.469154 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.469159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.471019 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.471030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.472538 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.472544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.472716 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.472749 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:09:28.473464 (+ 715us) client_negotiation.cc:240] Negotiation successful
0620 07:09:28.473593 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":1391,"threads_started":1}
I20260620 07:09:28.474848 8548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cce709dfff46b898b3dbc0328d5dc0" candidate_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c65855a594b4570b8139916901d4181" is_pre_election: true
I20260620 07:09:28.475315 8548 raft_consensus.cc:2454] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb51a1e996ce4d68b6f3f3c6791d266f in term 0.
I20260620 07:09:28.475975 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.466397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:54913 (local address 127.6.53.66:42645)
0620 07:09:28.466709 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:28.466719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:28.469277 (+ 2558us) server_negotiation.cc:407] Connection header received
0620 07:09:28.469440 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:28.469445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:28.469526 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:28.469610 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:28.470244 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.471389 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.473550 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.474786 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.474916 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.475016 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:28.475589 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 07:09:28.475726 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260620 07:09:28.476063 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.466102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:42645 (local address 127.6.53.65:54913)
0620 07:09:28.469165 (+ 3063us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:28.469182 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:28.469321 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:28.469644 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:28.469648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:28.469675 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:28.470063 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.470074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.471529 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.471535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.473345 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.473376 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.475175 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.475181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.475366 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.475401 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:09:28.475547 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 07:09:28.475685 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2918,"mutex_wait_us":352,"thread_start_us":2376,"threads_started":1}
I20260620 07:09:28.475955 8194 leader_election.cc:302] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c65855a594b4570b8139916901d4181, fb51a1e996ce4d68b6f3f3c6791d266f; no voters:
I20260620 07:09:28.476397 8620 raft_consensus.cc:2781] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:09:28.476524 8620 raft_consensus.cc:482] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:09:28.476626 8620 raft_consensus.cc:3037] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:28.476621 8407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cce709dfff46b898b3dbc0328d5dc0" candidate_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc24236063c4c7aa912060d76892563" is_pre_election: true
I20260620 07:09:28.476843 8407 raft_consensus.cc:2454] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb51a1e996ce4d68b6f3f3c6791d266f in term 0.
I20260620 07:09:28.479038 8620 raft_consensus.cc:504] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.479611 8620 leader_election.cc:288] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [CANDIDATE]: Term 1 election: Requested vote from peers 9dc24236063c4c7aa912060d76892563 (127.6.53.66:42645), 2c65855a594b4570b8139916901d4181 (127.6.53.67:38981)
I20260620 07:09:28.480012 8407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cce709dfff46b898b3dbc0328d5dc0" candidate_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc24236063c4c7aa912060d76892563"
I20260620 07:09:28.480132 8548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cce709dfff46b898b3dbc0328d5dc0" candidate_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c65855a594b4570b8139916901d4181"
I20260620 07:09:28.480247 8407 raft_consensus.cc:3037] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:28.480382 8548 raft_consensus.cc:3037] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:28.483264 8407 raft_consensus.cc:2454] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb51a1e996ce4d68b6f3f3c6791d266f in term 1.
I20260620 07:09:28.483429 8548 raft_consensus.cc:2454] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb51a1e996ce4d68b6f3f3c6791d266f in term 1.
I20260620 07:09:28.483721 8197 leader_election.cc:302] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dc24236063c4c7aa912060d76892563, fb51a1e996ce4d68b6f3f3c6791d266f; no voters:
I20260620 07:09:28.484058 8620 raft_consensus.cc:2781] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:28.484526 8620 raft_consensus.cc:686] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 LEADER]: Becoming Leader. State: Replica: fb51a1e996ce4d68b6f3f3c6791d266f, State: Running, Role: LEADER
I20260620 07:09:28.485211 8620 consensus_queue.cc:237] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.488415 8635 tablet_replica.cc:439] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f: TxnStatusTablet state changed. Reason: New leader fb51a1e996ce4d68b6f3f3c6791d266f. Latest consensus state: current_term: 1 leader_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } }
I20260620 07:09:28.488816 8635 tablet_replica.cc:442] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:09:28.489516 8637 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 07:09:28.489781 8637 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 07:09:28.489660 8107 catalog_manager.cc:5495] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f reported cstate change: term changed from 0 to 1, leader changed from <none> to fb51a1e996ce4d68b6f3f3c6791d266f (127.6.53.65). New cstate: current_term: 1 leader_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } health_report { overall_health: UNKNOWN } } }
I20260620 07:09:28.511113 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.505013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38753 (local address 127.0.0.1:54116)
0620 07:09:28.505228 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:28.505247 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:28.505367 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:28.505825 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:28.505828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:28.505837 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:09:28.505986 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.505992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.507451 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.507461 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.508536 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.508548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.509467 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.509471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.509616 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.509666 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:09:28.510646 (+ 980us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:09:28.510661 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 07:09:28.510717 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 07:09:28.510835 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 07:09:28.511179 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.505173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.6.53.65:38753)
0620 07:09:28.505369 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:28.505375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:28.505395 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:09:28.505533 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:28.505538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:28.505643 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:28.505794 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:28.506111 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.507281 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.508666 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.509311 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.509436 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.509840 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:28.510546 (+ 706us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:28.510684 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:28.510737 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 07:09:28.510818 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 07:09:28.519062 8548 raft_consensus.cc:1261] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181 [term 1 FOLLOWER]: Refusing update from remote peer fb51a1e996ce4d68b6f3f3c6791d266f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:28.519375 8407 raft_consensus.cc:1261] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Refusing update from remote peer fb51a1e996ce4d68b6f3f3c6791d266f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:28.520066 8635 consensus_queue.cc:1035] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:28.520386 8620 consensus_queue.cc:1035] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:28.527475 8619 tablet_replica.cc:439] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181: TxnStatusTablet state changed. Reason: New leader fb51a1e996ce4d68b6f3f3c6791d266f. Latest consensus state: current_term: 1 leader_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } }
I20260620 07:09:28.527884 8619 tablet_replica.cc:442] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:09:28.531132 8641 mvcc.cc:205] Tried to move back new op lower bound from 7298823653442596864 to 7298823653318623232. Current Snapshot: MvccSnapshot[applied={T|T < 7298823653442596864}]
I20260620 07:09:28.531791 8620 tablet_replica.cc:439] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } }
I20260620 07:09:28.532054 8620 tablet_replica.cc:442] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:09:28.532238 8640 mvcc.cc:205] Tried to move back new op lower bound from 7298823653442596864 to 7298823653318623232. Current Snapshot: MvccSnapshot[applied={T|T < 7298823653442596864}]
I20260620 07:09:28.540596 8619 tablet_replica.cc:439] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } }
I20260620 07:09:28.540887 8619 tablet_replica.cc:442] T 46cce709dfff46b898b3dbc0328d5dc0 P 2c65855a594b4570b8139916901d4181: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:09:28.545435 8618 tablet_replica.cc:439] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563: TxnStatusTablet state changed. Reason: New leader fb51a1e996ce4d68b6f3f3c6791d266f. Latest consensus state: current_term: 1 leader_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } }
I20260620 07:09:28.545748 8618 tablet_replica.cc:442] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:09:28.547048 8635 tablet_replica.cc:439] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } }
I20260620 07:09:28.547330 8635 tablet_replica.cc:442] T 46cce709dfff46b898b3dbc0328d5dc0 P fb51a1e996ce4d68b6f3f3c6791d266f: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:09:28.550772 8618 tablet_replica.cc:439] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } } }
I20260620 07:09:28.551025 8618 tablet_replica.cc:442] T 46cce709dfff46b898b3dbc0328d5dc0 P 9dc24236063c4c7aa912060d76892563: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:09:28.551595 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.542726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42762 (local address 127.6.53.67:38981)
0620 07:09:28.542901 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:28.542908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:28.543126 (+ 218us) server_negotiation.cc:407] Connection header received
0620 07:09:28.543368 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:28.543372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:28.543463 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:28.543618 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:28.544211 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.545276 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.547202 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.547810 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.547920 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.549926 (+ 2006us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:28.550598 (+ 672us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:28.550739 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:28.551232 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 07:09:28.551310 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:09:28.566722 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.561359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38753 (local address 127.6.53.67:39961)
0620 07:09:28.561639 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:28.561655 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:28.561751 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:28.562123 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:28.562126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:28.562136 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:09:28.562324 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.562332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.563753 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.563756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.564716 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.564726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.565693 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.565697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.565803 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.565849 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:09:28.566329 (+ 480us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:09:28.566337 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:09:28.566401 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 07:09:28.566501 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260620 07:09:28.566807 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.561336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:39961 (local address 127.6.53.65:38753)
0620 07:09:28.561511 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:28.561517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:28.561721 (+ 204us) server_negotiation.cc:407] Connection header received
0620 07:09:28.561880 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:28.561886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:28.562018 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:28.562116 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:28.562468 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.563627 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.564846 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.565550 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.565681 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.566014 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:28.566165 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:28.566285 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:28.566448 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 07:09:28.566521 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:09:28.575836 8406 raft_consensus.cc:1261] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Refusing update from remote peer 2c65855a594b4570b8139916901d4181: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:28.576366 8269 raft_consensus.cc:1261] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Refusing update from remote peer 2c65855a594b4570b8139916901d4181: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:28.576568 8622 consensus_queue.cc:1035] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:28.577039 8619 consensus_queue.cc:1035] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:28.629261 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.618779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:41685 (local address 127.6.53.67:38981)
0620 07:09:28.618949 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:28.618955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:28.622938 (+ 3983us) server_negotiation.cc:407] Connection header received
0620 07:09:28.623125 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:28.623130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:28.623211 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:28.623318 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:28.623932 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.624981 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.627191 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.627855 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.627991 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.628518 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:28.628654 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:28.628748 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:28.628885 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 07:09:28.628971 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:09:28.629405 8654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:28.618674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:38981 (local address 127.6.53.65:41685)
0620 07:09:28.622819 (+ 4145us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:28.622840 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:28.622985 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:28.623354 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:28.623360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:28.623379 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:09:28.623753 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.623765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.625140 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.625146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.627029 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:28.627044 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:28.628051 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:28.628057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:28.628255 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:28.628322 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:09:28.628759 (+ 437us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:09:28.628770 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 07:09:28.628848 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 07:09:28.628981 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3935,"thread_start_us":3789,"threads_started":1}
I20260620 07:09:28.672607 8105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:09:28.672993 8105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:09:28.677313 8105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9: Sending DeleteTablet for 3 replicas of tablet 13395f9492d04b238bd55f36f9207e3f
I20260620 07:09:28.678401 8527 tablet_service.cc:1514] Processing DeleteTablet for tablet 13395f9492d04b238bd55f36f9207e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:28 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:42754
I20260620 07:09:28.678439 8244 tablet_service.cc:1514] Processing DeleteTablet for tablet 13395f9492d04b238bd55f36f9207e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:28 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:54110
I20260620 07:09:28.680282 8383 tablet_service.cc:1514] Processing DeleteTablet for tablet 13395f9492d04b238bd55f36f9207e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:28 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:49238
I20260620 07:09:28.681232 8107 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:09:28.681830 8107 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:09:28.690151 8382 tablet_service.cc:1467] Processing CreateTablet for tablet c6d5c1028f004f7dba9cac9f22815f6f (DEFAULT_TABLE table=test-table [id=81a03ca8582942efafe07135f0e7bf2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:28.690477 8243 tablet_service.cc:1467] Processing CreateTablet for tablet c6d5c1028f004f7dba9cac9f22815f6f (DEFAULT_TABLE table=test-table [id=81a03ca8582942efafe07135f0e7bf2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:28.690758 8660 tablet_replica.cc:330] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f: stopping tablet replica
I20260620 07:09:28.690878 8382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d5c1028f004f7dba9cac9f22815f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:28.691072 8243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d5c1028f004f7dba9cac9f22815f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:28.691262 8660 raft_consensus.cc:2229] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:09:28.691613 8660 raft_consensus.cc:2258] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:28.693699 8660 ts_tablet_manager.cc:1905] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:28.694973 8526 tablet_service.cc:1467] Processing CreateTablet for tablet c6d5c1028f004f7dba9cac9f22815f6f (DEFAULT_TABLE table=test-table [id=81a03ca8582942efafe07135f0e7bf2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:28.695618 8526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d5c1028f004f7dba9cac9f22815f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:28.700727 8660 ts_tablet_manager.cc:1918] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:09:28.700903 8660 log.cc:1196] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/13395f9492d04b238bd55f36f9207e3f
I20260620 07:09:28.701315 8614 tablet_bootstrap.cc:492] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f: Bootstrap starting.
I20260620 07:09:28.701429 8660 ts_tablet_manager.cc:1939] T 13395f9492d04b238bd55f36f9207e3f P fb51a1e996ce4d68b6f3f3c6791d266f: Deleting consensus metadata
I20260620 07:09:28.703186 8614 tablet_bootstrap.cc:654] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:28.705574 8614 tablet_bootstrap.cc:492] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f: No bootstrap required, opened a new log
I20260620 07:09:28.705797 8614 ts_tablet_manager.cc:1397] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 07:09:28.706660 8614 raft_consensus.cc:348] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.707022 8614 raft_consensus.cc:374] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:28.707127 8614 raft_consensus.cc:729] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb51a1e996ce4d68b6f3f3c6791d266f, State: Initialized, Role: FOLLOWER
I20260620 07:09:28.707448 8613 tablet_bootstrap.cc:492] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563: Bootstrap starting.
I20260620 07:09:28.707432 8614 consensus_queue.cc:260] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.708302 8614 ts_tablet_manager.cc:1428] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:09:28.709184 8661 tablet_replica.cc:330] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563: stopping tablet replica
I20260620 07:09:28.709295 8613 tablet_bootstrap.cc:654] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:28.709679 8661 raft_consensus.cc:2229] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:09:28.710100 8661 raft_consensus.cc:2258] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:28.713261 8659 tablet_replica.cc:330] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181: stopping tablet replica
I20260620 07:09:28.713351 8612 tablet_bootstrap.cc:492] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181: Bootstrap starting.
I20260620 07:09:28.713712 8659 raft_consensus.cc:2229] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:09:28.714481 8659 raft_consensus.cc:2258] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:28.715240 8612 tablet_bootstrap.cc:654] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:28.715494 8093 catalog_manager.cc:4841] TS fb51a1e996ce4d68b6f3f3c6791d266f (127.6.53.65:38753): tablet 13395f9492d04b238bd55f36f9207e3f (table test-table [id=aaa3bd6321694612a0451c4a88f8bca9]) successfully deleted
I20260620 07:09:28.715899 8661 ts_tablet_manager.cc:1905] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:28.717473 8659 ts_tablet_manager.cc:1905] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:28.724871 8661 ts_tablet_manager.cc:1918] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:09:28.725097 8661 log.cc:1196] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/13395f9492d04b238bd55f36f9207e3f
I20260620 07:09:28.725080 8659 ts_tablet_manager.cc:1918] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:09:28.725296 8659 log.cc:1196] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/13395f9492d04b238bd55f36f9207e3f
I20260620 07:09:28.725848 8661 ts_tablet_manager.cc:1939] T 13395f9492d04b238bd55f36f9207e3f P 9dc24236063c4c7aa912060d76892563: Deleting consensus metadata
I20260620 07:09:28.726022 8659 ts_tablet_manager.cc:1939] T 13395f9492d04b238bd55f36f9207e3f P 2c65855a594b4570b8139916901d4181: Deleting consensus metadata
I20260620 07:09:28.727653 8094 catalog_manager.cc:4841] TS 9dc24236063c4c7aa912060d76892563 (127.6.53.66:42645): tablet 13395f9492d04b238bd55f36f9207e3f (table test-table [id=aaa3bd6321694612a0451c4a88f8bca9]) successfully deleted
I20260620 07:09:28.727842 8091 catalog_manager.cc:4841] TS 2c65855a594b4570b8139916901d4181 (127.6.53.67:38981): tablet 13395f9492d04b238bd55f36f9207e3f (table test-table [id=aaa3bd6321694612a0451c4a88f8bca9]) successfully deleted
I20260620 07:09:28.728988 8613 tablet_bootstrap.cc:492] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563: No bootstrap required, opened a new log
I20260620 07:09:28.729163 8613 ts_tablet_manager.cc:1397] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.008s
I20260620 07:09:28.729490 8612 tablet_bootstrap.cc:492] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181: No bootstrap required, opened a new log
I20260620 07:09:28.729655 8612 ts_tablet_manager.cc:1397] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20260620 07:09:28.730026 8613 raft_consensus.cc:348] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.730309 8613 raft_consensus.cc:374] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:28.730408 8613 raft_consensus.cc:729] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc24236063c4c7aa912060d76892563, State: Initialized, Role: FOLLOWER
I20260620 07:09:28.730525 8612 raft_consensus.cc:348] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.730777 8612 raft_consensus.cc:374] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:28.730667 8613 consensus_queue.cc:260] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.730862 8612 raft_consensus.cc:729] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c65855a594b4570b8139916901d4181, State: Initialized, Role: FOLLOWER
I20260620 07:09:28.731338 8613 ts_tablet_manager.cc:1428] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:09:28.731163 8612 consensus_queue.cc:260] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.735121 8612 ts_tablet_manager.cc:1428] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260620 07:09:28.764449 8655 raft_consensus.cc:482] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:09:28.764676 8655 raft_consensus.cc:504] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.765501 8655 leader_election.cc:288] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb51a1e996ce4d68b6f3f3c6791d266f (127.6.53.65:38753), 9dc24236063c4c7aa912060d76892563 (127.6.53.66:42645)
I20260620 07:09:28.766094 8269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d5c1028f004f7dba9cac9f22815f6f" candidate_uuid: "2c65855a594b4570b8139916901d4181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" is_pre_election: true
I20260620 07:09:28.766147 8406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d5c1028f004f7dba9cac9f22815f6f" candidate_uuid: "2c65855a594b4570b8139916901d4181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc24236063c4c7aa912060d76892563" is_pre_election: true
I20260620 07:09:28.766438 8269 raft_consensus.cc:2454] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c65855a594b4570b8139916901d4181 in term 0.
I20260620 07:09:28.766541 8406 raft_consensus.cc:2454] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c65855a594b4570b8139916901d4181 in term 0.
I20260620 07:09:28.766959 8475 leader_election.cc:302] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c65855a594b4570b8139916901d4181, fb51a1e996ce4d68b6f3f3c6791d266f; no voters:
I20260620 07:09:28.767477 8655 raft_consensus.cc:2781] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:09:28.767611 8655 raft_consensus.cc:482] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:09:28.767714 8655 raft_consensus.cc:3037] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:28.771005 8655 raft_consensus.cc:504] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.772324 8406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d5c1028f004f7dba9cac9f22815f6f" candidate_uuid: "2c65855a594b4570b8139916901d4181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc24236063c4c7aa912060d76892563"
I20260620 07:09:28.772360 8269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d5c1028f004f7dba9cac9f22815f6f" candidate_uuid: "2c65855a594b4570b8139916901d4181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f"
I20260620 07:09:28.772553 8406 raft_consensus.cc:3037] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:28.772579 8269 raft_consensus.cc:3037] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:28.775600 8406 raft_consensus.cc:2454] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c65855a594b4570b8139916901d4181 in term 1.
I20260620 07:09:28.776516 8476 leader_election.cc:302] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c65855a594b4570b8139916901d4181, 9dc24236063c4c7aa912060d76892563; no voters:
I20260620 07:09:28.776796 8622 raft_consensus.cc:2781] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:28.777024 8622 raft_consensus.cc:686] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [term 1 LEADER]: Becoming Leader. State: Replica: 2c65855a594b4570b8139916901d4181, State: Running, Role: LEADER
I20260620 07:09:28.775851 8269 raft_consensus.cc:2454] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c65855a594b4570b8139916901d4181 in term 1.
I20260620 07:09:28.777284 8622 consensus_queue.cc:237] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:28.777848 8655 leader_election.cc:288] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [CANDIDATE]: Term 1 election: Requested vote from peers fb51a1e996ce4d68b6f3f3c6791d266f (127.6.53.65:38753), 9dc24236063c4c7aa912060d76892563 (127.6.53.66:42645)
I20260620 07:09:28.780911 8105 catalog_manager.cc:5495] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c65855a594b4570b8139916901d4181 (127.6.53.67). New cstate: current_term: 1 leader_uuid: "2c65855a594b4570b8139916901d4181" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } health_report { overall_health: HEALTHY } } }
I20260620 07:09:28.824357 8269 raft_consensus.cc:1261] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Refusing update from remote peer 2c65855a594b4570b8139916901d4181: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:28.825043 8655 consensus_queue.cc:1035] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:28.826095 8406 raft_consensus.cc:1261] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Refusing update from remote peer 2c65855a594b4570b8139916901d4181: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:28.827922 8622 consensus_queue.cc:1035] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:09:28.852751 8641 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 07:09:30.449342 8107 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:09:30.449599 8107 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:09:30.453399 8107 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9: Sending DeleteTablet for 3 replicas of tablet c6d5c1028f004f7dba9cac9f22815f6f
I20260620 07:09:30.454173 8243 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d5c1028f004f7dba9cac9f22815f6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:30 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:54110
I20260620 07:09:30.454177 8382 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d5c1028f004f7dba9cac9f22815f6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:30 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:49238
I20260620 07:09:30.455102 8690 tablet_replica.cc:330] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f: stopping tablet replica
I20260620 07:09:30.455124 8526 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d5c1028f004f7dba9cac9f22815f6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:30 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:42754
I20260620 07:09:30.455338 8690 raft_consensus.cc:2229] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:09:30.455499 8691 tablet_replica.cc:330] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563: stopping tablet replica
I20260620 07:09:30.455606 8690 raft_consensus.cc:2258] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:30.455724 8691 raft_consensus.cc:2229] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:09:30.455986 8691 raft_consensus.cc:2258] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:30.456535 8692 tablet_replica.cc:330] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181: stopping tablet replica
I20260620 07:09:30.456754 8692 raft_consensus.cc:2229] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:09:30.457118 8692 raft_consensus.cc:2258] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:30.458055 8692 ts_tablet_manager.cc:1905] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:30.459107 8691 ts_tablet_manager.cc:1905] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:30.461305 8690 ts_tablet_manager.cc:1905] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:30.466086 8691 ts_tablet_manager.cc:1918] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:09:30.466241 8691 log.cc:1196] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/c6d5c1028f004f7dba9cac9f22815f6f
I20260620 07:09:30.466666 8691 ts_tablet_manager.cc:1939] T c6d5c1028f004f7dba9cac9f22815f6f P 9dc24236063c4c7aa912060d76892563: Deleting consensus metadata
I20260620 07:09:30.467710 8094 catalog_manager.cc:4841] TS 9dc24236063c4c7aa912060d76892563 (127.6.53.66:42645): tablet c6d5c1028f004f7dba9cac9f22815f6f (table test-table [id=81a03ca8582942efafe07135f0e7bf2c]) successfully deleted
I20260620 07:09:30.469185 8690 ts_tablet_manager.cc:1918] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:09:30.469338 8690 log.cc:1196] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/c6d5c1028f004f7dba9cac9f22815f6f
I20260620 07:09:30.469813 8690 ts_tablet_manager.cc:1939] T c6d5c1028f004f7dba9cac9f22815f6f P fb51a1e996ce4d68b6f3f3c6791d266f: Deleting consensus metadata
I20260620 07:09:30.470993 8093 catalog_manager.cc:4841] TS fb51a1e996ce4d68b6f3f3c6791d266f (127.6.53.65:38753): tablet c6d5c1028f004f7dba9cac9f22815f6f (table test-table [id=81a03ca8582942efafe07135f0e7bf2c]) successfully deleted
I20260620 07:09:30.472792 8692 ts_tablet_manager.cc:1918] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:09:30.472915 8692 log.cc:1196] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/c6d5c1028f004f7dba9cac9f22815f6f
I20260620 07:09:30.473285 8692 ts_tablet_manager.cc:1939] T c6d5c1028f004f7dba9cac9f22815f6f P 2c65855a594b4570b8139916901d4181: Deleting consensus metadata
I20260620 07:09:30.474274 8091 catalog_manager.cc:4841] TS 2c65855a594b4570b8139916901d4181 (127.6.53.67:38981): tablet c6d5c1028f004f7dba9cac9f22815f6f (table test-table [id=81a03ca8582942efafe07135f0e7bf2c]) successfully deleted
Jun 20 07:09:30 dist-test-slave-4n6h krb5kdc[8062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939370, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:09:30 dist-test-slave-4n6h krb5kdc[8062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939370, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:30.505913 8700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:30.492024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48880 (local address 127.6.53.126:43215)
0620 07:09:30.492549 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:30.492553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:30.492576 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:09:30.492698 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:30.492702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:30.492762 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:30.492856 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:30.494141 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.495014 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.496361 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.496803 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.496897 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:30.500796 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:30.500801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:30.500805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:30.500842 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:30.503732 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:30.504373 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:30.504377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:30.504380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:30.504545 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:30.504860 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:30.504863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:30.504865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:30.505128 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:30.505299 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:30.505492 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 07:09:30.505645 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":279,"threads_started":1}
I20260620 07:09:30.509285 8107 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:09:30.509902 8107 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:09:30.516093 8243 tablet_service.cc:1467] Processing CreateTablet for tablet e8fb972f5e1e4906bb25974e40b305dc (DEFAULT_TABLE table=test-table [id=ddcac10796f74797800a225bd2eac162]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:30.516317 8382 tablet_service.cc:1467] Processing CreateTablet for tablet e8fb972f5e1e4906bb25974e40b305dc (DEFAULT_TABLE table=test-table [id=ddcac10796f74797800a225bd2eac162]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:30.516631 8243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8fb972f5e1e4906bb25974e40b305dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:30.516574 8526 tablet_service.cc:1467] Processing CreateTablet for tablet e8fb972f5e1e4906bb25974e40b305dc (DEFAULT_TABLE table=test-table [id=ddcac10796f74797800a225bd2eac162]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:30.516746 8382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8fb972f5e1e4906bb25974e40b305dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:30.516983 8526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8fb972f5e1e4906bb25974e40b305dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:30.521296 8701 tablet_bootstrap.cc:492] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f: Bootstrap starting.
I20260620 07:09:30.522868 8701 tablet_bootstrap.cc:654] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:30.523635 8702 tablet_bootstrap.cc:492] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181: Bootstrap starting.
I20260620 07:09:30.524405 8703 tablet_bootstrap.cc:492] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563: Bootstrap starting.
I20260620 07:09:30.525961 8703 tablet_bootstrap.cc:654] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:30.526544 8702 tablet_bootstrap.cc:654] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:30.527598 8701 tablet_bootstrap.cc:492] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f: No bootstrap required, opened a new log
I20260620 07:09:30.527797 8701 ts_tablet_manager.cc:1397] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260620 07:09:30.528837 8702 tablet_bootstrap.cc:492] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181: No bootstrap required, opened a new log
I20260620 07:09:30.528698 8701 raft_consensus.cc:348] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:30.528941 8701 raft_consensus.cc:374] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:30.529001 8702 ts_tablet_manager.cc:1397] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 07:09:30.529050 8701 raft_consensus.cc:729] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb51a1e996ce4d68b6f3f3c6791d266f, State: Initialized, Role: FOLLOWER
I20260620 07:09:30.529157 8703 tablet_bootstrap.cc:492] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563: No bootstrap required, opened a new log
I20260620 07:09:30.529279 8703 ts_tablet_manager.cc:1397] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 07:09:30.529300 8701 consensus_queue.cc:260] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:30.529903 8701 ts_tablet_manager.cc:1428] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 07:09:30.529832 8702 raft_consensus.cc:348] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:30.530064 8702 raft_consensus.cc:374] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:30.530154 8702 raft_consensus.cc:729] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c65855a594b4570b8139916901d4181, State: Initialized, Role: FOLLOWER
I20260620 07:09:30.530059 8703 raft_consensus.cc:348] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:30.530342 8703 raft_consensus.cc:374] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:30.530452 8703 raft_consensus.cc:729] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc24236063c4c7aa912060d76892563, State: Initialized, Role: FOLLOWER
I20260620 07:09:30.530423 8702 consensus_queue.cc:260] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:30.530941 8702 ts_tablet_manager.cc:1428] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:09:30.530719 8703 consensus_queue.cc:260] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:30.534781 8703 ts_tablet_manager.cc:1428] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260620 07:09:30.779572 8707 raft_consensus.cc:482] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:09:30.779855 8707 raft_consensus.cc:504] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:30.781463 8707 leader_election.cc:288] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb51a1e996ce4d68b6f3f3c6791d266f (127.6.53.65:38753), 2c65855a594b4570b8139916901d4181 (127.6.53.67:38981)
I20260620 07:09:30.798951 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:30.781961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:44717 (local address 127.6.53.67:38981)
0620 07:09:30.792422 (+ 10461us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:30.792432 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:30.792945 (+ 513us) server_negotiation.cc:407] Connection header received
0620 07:09:30.793137 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:30.793143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:30.793228 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:30.793348 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:30.794028 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.795016 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.796987 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.797857 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.798042 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:30.798189 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:30.798474 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 07:09:30.798628 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10236,"thread_start_us":10054,"threads_started":1}
I20260620 07:09:30.798998 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:30.781748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:38981 (local address 127.6.53.66:44717)
0620 07:09:30.792731 (+ 10983us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:30.792755 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:30.792960 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:30.793399 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:30.793404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:30.793457 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:30.793804 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:30.793822 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.795156 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.795161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:30.796828 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:30.796840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.798091 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.798098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:30.798291 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:30.798331 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 07:09:30.798447 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 07:09:30.798587 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10811,"mutex_wait_us":1,"thread_start_us":11378,"threads_started":1}
I20260620 07:09:30.799439 8708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:30.781782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38753 (local address 127.6.53.66:59781)
0620 07:09:30.793233 (+ 11451us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:30.793250 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:30.793377 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:30.793725 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:30.793728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:30.793750 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:30.794212 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:30.794226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.795441 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.795445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:30.797074 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:30.797088 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.798387 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.798392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:30.798527 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:30.798564 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 07:09:30.798672 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 07:09:30.798797 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11312,"mutex_wait_us":8,"thread_start_us":10677,"threads_started":1}
I20260620 07:09:30.799494 8709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:30.781785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:59781 (local address 127.6.53.65:38753)
0620 07:09:30.793106 (+ 11321us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:30.793112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:30.793357 (+ 245us) server_negotiation.cc:407] Connection header received
0620 07:09:30.793509 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:30.793515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:30.793593 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:30.793705 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:30.794402 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.795238 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.797224 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.798092 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.798213 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:30.798354 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:30.798698 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 07:09:30.798832 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11176,"thread_start_us":11021,"threads_started":1}
I20260620 07:09:30.799852 8548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8fb972f5e1e4906bb25974e40b305dc" candidate_uuid: "9dc24236063c4c7aa912060d76892563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c65855a594b4570b8139916901d4181" is_pre_election: true
I20260620 07:09:30.800127 8548 raft_consensus.cc:2454] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dc24236063c4c7aa912060d76892563 in term 0.
I20260620 07:09:30.800174 8269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8fb972f5e1e4906bb25974e40b305dc" candidate_uuid: "9dc24236063c4c7aa912060d76892563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" is_pre_election: true
I20260620 07:09:30.800431 8269 raft_consensus.cc:2454] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dc24236063c4c7aa912060d76892563 in term 0.
I20260620 07:09:30.800740 8334 leader_election.cc:302] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c65855a594b4570b8139916901d4181, 9dc24236063c4c7aa912060d76892563; no voters:
I20260620 07:09:30.801178 8707 raft_consensus.cc:2781] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:09:30.801306 8707 raft_consensus.cc:482] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:09:30.801416 8707 raft_consensus.cc:3037] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:30.803833 8707 raft_consensus.cc:504] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:30.804378 8707 leader_election.cc:288] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [CANDIDATE]: Term 1 election: Requested vote from peers fb51a1e996ce4d68b6f3f3c6791d266f (127.6.53.65:38753), 2c65855a594b4570b8139916901d4181 (127.6.53.67:38981)
I20260620 07:09:30.804850 8548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8fb972f5e1e4906bb25974e40b305dc" candidate_uuid: "9dc24236063c4c7aa912060d76892563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c65855a594b4570b8139916901d4181"
I20260620 07:09:30.804850 8269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8fb972f5e1e4906bb25974e40b305dc" candidate_uuid: "9dc24236063c4c7aa912060d76892563" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f"
I20260620 07:09:30.805053 8548 raft_consensus.cc:3037] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:30.805056 8269 raft_consensus.cc:3037] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:30.808037 8269 raft_consensus.cc:2454] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dc24236063c4c7aa912060d76892563 in term 1.
I20260620 07:09:30.808037 8548 raft_consensus.cc:2454] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dc24236063c4c7aa912060d76892563 in term 1.
I20260620 07:09:30.808418 8334 leader_election.cc:302] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c65855a594b4570b8139916901d4181, 9dc24236063c4c7aa912060d76892563; no voters:
I20260620 07:09:30.808697 8707 raft_consensus.cc:2781] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:30.809130 8707 raft_consensus.cc:686] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [term 1 LEADER]: Becoming Leader. State: Replica: 9dc24236063c4c7aa912060d76892563, State: Running, Role: LEADER
I20260620 07:09:30.809543 8707 consensus_queue.cc:237] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:30.812757 8107 catalog_manager.cc:5495] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dc24236063c4c7aa912060d76892563 (127.6.53.66). New cstate: current_term: 1 leader_uuid: "9dc24236063c4c7aa912060d76892563" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } health_report { overall_health: UNKNOWN } } }
I20260620 07:09:30.907390 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:30.898689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.6.53.66:42645)
0620 07:09:30.902309 (+ 3620us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:30.902313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:30.902334 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:09:30.902412 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:30.902416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:30.902488 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:30.902625 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:30.903014 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.903828 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.905200 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.905619 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.905714 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:30.906128 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:30.906742 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:30.906895 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:30.907058 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 07:09:30.907122 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3484,"thread_start_us":3367,"threads_started":1}
I20260620 07:09:30.922324 8709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:30.917367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:59035 (local address 127.6.53.65:38753)
0620 07:09:30.917586 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:30.917594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:30.918229 (+ 635us) server_negotiation.cc:407] Connection header received
0620 07:09:30.918482 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:30.918486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:30.918548 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:30.918622 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:30.919049 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.919702 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.920941 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.921312 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.921398 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:30.921752 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:30.921843 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:30.921923 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:30.922041 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 07:09:30.922095 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:09:30.922456 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:30.917304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38753 (local address 127.6.53.66:59035)
0620 07:09:30.918112 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:30.918134 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:30.918316 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:30.918651 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:30.918654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:30.918663 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:09:30.918896 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:30.918902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.919806 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.919809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:30.920822 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:30.920832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.921445 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.921449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:30.921561 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:30.921606 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:09:30.921941 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:09:30.921949 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:09:30.922008 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 07:09:30.922098 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":375,"threads_started":1}
I20260620 07:09:30.930323 8269 raft_consensus.cc:1261] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Refusing update from remote peer 9dc24236063c4c7aa912060d76892563: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:30.930323 8548 raft_consensus.cc:1261] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181 [term 1 FOLLOWER]: Refusing update from remote peer 9dc24236063c4c7aa912060d76892563: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:30.930929 8707 consensus_queue.cc:1035] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:30.931231 8712 consensus_queue.cc:1035] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:30.957013 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:30.951703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:33411 (local address 127.6.53.66:42645)
0620 07:09:30.951868 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:30.951876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:30.952180 (+ 304us) server_negotiation.cc:407] Connection header received
0620 07:09:30.952394 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:30.952398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:30.952456 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:30.952525 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:30.952862 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.953573 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.954979 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.955613 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.955736 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:30.956261 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:30.956417 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:30.956512 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:30.956703 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 07:09:30.956782 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 07:09:30.957157 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:30.951447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:42645 (local address 127.6.53.65:33411)
0620 07:09:30.952083 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:30.952100 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:30.952248 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:30.952541 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:30.952544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:30.952553 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:09:30.952746 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:30.952753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.953679 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.953683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:30.954870 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:30.954882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:30.955789 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:30.955795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:30.956023 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:30.956071 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:09:30.956532 (+ 461us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:09:30.956542 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 07:09:30.956659 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 07:09:30.956820 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":455,"threads_started":1}
I20260620 07:09:30.987043 8105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:09:30.987231 8105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:09:30.990070 8105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9: Sending DeleteTablet for 3 replicas of tablet e8fb972f5e1e4906bb25974e40b305dc
I20260620 07:09:30.990782 8243 tablet_service.cc:1514] Processing DeleteTablet for tablet e8fb972f5e1e4906bb25974e40b305dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:30 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:54110
I20260620 07:09:30.990931 8382 tablet_service.cc:1514] Processing DeleteTablet for tablet e8fb972f5e1e4906bb25974e40b305dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:30 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:49238
I20260620 07:09:30.991225 8526 tablet_service.cc:1514] Processing DeleteTablet for tablet e8fb972f5e1e4906bb25974e40b305dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:30 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:42754
I20260620 07:09:30.992981 8105 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:09:30.994505 8105 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:09:31.003752 8726 tablet_replica.cc:330] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f: stopping tablet replica
I20260620 07:09:31.003964 8726 raft_consensus.cc:2229] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:09:31.004040 8244 tablet_service.cc:1467] Processing CreateTablet for tablet 8b8b5877cfaa4799896c2f106d238c9e (DEFAULT_TABLE table=test-table [id=6f5295f3688944b185043e53d9834b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:31.004184 8726 raft_consensus.cc:2258] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:31.004442 8244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8b5877cfaa4799896c2f106d238c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:31.004674 8383 tablet_service.cc:1467] Processing CreateTablet for tablet 8b8b5877cfaa4799896c2f106d238c9e (DEFAULT_TABLE table=test-table [id=6f5295f3688944b185043e53d9834b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:31.005120 8383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8b5877cfaa4799896c2f106d238c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:31.011226 8527 tablet_service.cc:1467] Processing CreateTablet for tablet 8b8b5877cfaa4799896c2f106d238c9e (DEFAULT_TABLE table=test-table [id=6f5295f3688944b185043e53d9834b4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:31.011653 8527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8b5877cfaa4799896c2f106d238c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:31.013535 8728 tablet_replica.cc:330] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181: stopping tablet replica
I20260620 07:09:31.014487 8701 tablet_bootstrap.cc:492] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f: Bootstrap starting.
I20260620 07:09:31.014833 8728 raft_consensus.cc:2229] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:09:31.015122 8728 raft_consensus.cc:2258] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:31.015727 8727 tablet_replica.cc:330] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563: stopping tablet replica
I20260620 07:09:31.015926 8727 raft_consensus.cc:2229] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:09:31.016175 8701 tablet_bootstrap.cc:654] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:31.017940 8728 ts_tablet_manager.cc:1905] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:31.018383 8727 raft_consensus.cc:2258] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:31.019547 8727 ts_tablet_manager.cc:1905] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:31.020941 8702 tablet_bootstrap.cc:492] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181: Bootstrap starting.
I20260620 07:09:31.022614 8702 tablet_bootstrap.cc:654] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:31.022842 8703 tablet_bootstrap.cc:492] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563: Bootstrap starting.
I20260620 07:09:31.023825 8701 tablet_bootstrap.cc:492] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f: No bootstrap required, opened a new log
I20260620 07:09:31.023995 8701 ts_tablet_manager.cc:1397] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20260620 07:09:31.024549 8703 tablet_bootstrap.cc:654] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:31.024834 8701 raft_consensus.cc:348] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:31.025103 8701 raft_consensus.cc:374] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:31.025198 8701 raft_consensus.cc:729] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb51a1e996ce4d68b6f3f3c6791d266f, State: Initialized, Role: FOLLOWER
I20260620 07:09:31.025501 8701 consensus_queue.cc:260] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:31.026652 8701 ts_tablet_manager.cc:1428] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 07:09:31.027845 8726 ts_tablet_manager.cc:1905] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:31.028381 8727 ts_tablet_manager.cc:1918] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:09:31.028586 8727 log.cc:1196] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/e8fb972f5e1e4906bb25974e40b305dc
I20260620 07:09:31.029145 8727 ts_tablet_manager.cc:1939] T e8fb972f5e1e4906bb25974e40b305dc P 9dc24236063c4c7aa912060d76892563: Deleting consensus metadata
I20260620 07:09:31.030242 8094 catalog_manager.cc:4841] TS 9dc24236063c4c7aa912060d76892563 (127.6.53.66:42645): tablet e8fb972f5e1e4906bb25974e40b305dc (table test-table [id=ddcac10796f74797800a225bd2eac162]) successfully deleted
I20260620 07:09:31.035899 8726 ts_tablet_manager.cc:1918] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:09:31.036024 8726 log.cc:1196] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/e8fb972f5e1e4906bb25974e40b305dc
I20260620 07:09:31.036442 8726 ts_tablet_manager.cc:1939] T e8fb972f5e1e4906bb25974e40b305dc P fb51a1e996ce4d68b6f3f3c6791d266f: Deleting consensus metadata
I20260620 07:09:31.036968 8728 ts_tablet_manager.cc:1918] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:09:31.037099 8728 log.cc:1196] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/e8fb972f5e1e4906bb25974e40b305dc
I20260620 07:09:31.037397 8093 catalog_manager.cc:4841] TS fb51a1e996ce4d68b6f3f3c6791d266f (127.6.53.65:38753): tablet e8fb972f5e1e4906bb25974e40b305dc (table test-table [id=ddcac10796f74797800a225bd2eac162]) successfully deleted
I20260620 07:09:31.037598 8728 ts_tablet_manager.cc:1939] T e8fb972f5e1e4906bb25974e40b305dc P 2c65855a594b4570b8139916901d4181: Deleting consensus metadata
I20260620 07:09:31.038915 8091 catalog_manager.cc:4841] TS 2c65855a594b4570b8139916901d4181 (127.6.53.67:38981): tablet e8fb972f5e1e4906bb25974e40b305dc (table test-table [id=ddcac10796f74797800a225bd2eac162]) successfully deleted
I20260620 07:09:31.043267 8703 tablet_bootstrap.cc:492] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563: No bootstrap required, opened a new log
I20260620 07:09:31.043318 8702 tablet_bootstrap.cc:492] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181: No bootstrap required, opened a new log
I20260620 07:09:31.043478 8703 ts_tablet_manager.cc:1397] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.004s
I20260620 07:09:31.043484 8702 ts_tablet_manager.cc:1397] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.000s
I20260620 07:09:31.044092 8703 raft_consensus.cc:348] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:31.044319 8703 raft_consensus.cc:374] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:31.044224 8702 raft_consensus.cc:348] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:31.044406 8703 raft_consensus.cc:729] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc24236063c4c7aa912060d76892563, State: Initialized, Role: FOLLOWER
I20260620 07:09:31.044515 8702 raft_consensus.cc:374] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:31.044605 8702 raft_consensus.cc:729] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c65855a594b4570b8139916901d4181, State: Initialized, Role: FOLLOWER
I20260620 07:09:31.044706 8703 consensus_queue.cc:260] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:31.044831 8702 consensus_queue.cc:260] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:31.045261 8703 ts_tablet_manager.cc:1428] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 07:09:31.045368 8702 ts_tablet_manager.cc:1428] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:09:31.154026 8620 raft_consensus.cc:482] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:09:31.154237 8620 raft_consensus.cc:504] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:31.154996 8620 leader_election.cc:288] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dc24236063c4c7aa912060d76892563 (127.6.53.66:42645), 2c65855a594b4570b8139916901d4181 (127.6.53.67:38981)
I20260620 07:09:31.155524 8406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8b5877cfaa4799896c2f106d238c9e" candidate_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc24236063c4c7aa912060d76892563" is_pre_election: true
I20260620 07:09:31.155570 8548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8b5877cfaa4799896c2f106d238c9e" candidate_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c65855a594b4570b8139916901d4181" is_pre_election: true
I20260620 07:09:31.155784 8548 raft_consensus.cc:2454] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb51a1e996ce4d68b6f3f3c6791d266f in term 0.
I20260620 07:09:31.155812 8406 raft_consensus.cc:2454] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb51a1e996ce4d68b6f3f3c6791d266f in term 0.
I20260620 07:09:31.156219 8194 leader_election.cc:302] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c65855a594b4570b8139916901d4181, 9dc24236063c4c7aa912060d76892563, fb51a1e996ce4d68b6f3f3c6791d266f; no voters:
I20260620 07:09:31.156535 8620 raft_consensus.cc:2781] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:09:31.156665 8620 raft_consensus.cc:482] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:09:31.156756 8620 raft_consensus.cc:3037] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:31.159736 8620 raft_consensus.cc:504] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:31.160356 8620 leader_election.cc:288] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [CANDIDATE]: Term 1 election: Requested vote from peers 9dc24236063c4c7aa912060d76892563 (127.6.53.66:42645), 2c65855a594b4570b8139916901d4181 (127.6.53.67:38981)
I20260620 07:09:31.160867 8406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8b5877cfaa4799896c2f106d238c9e" candidate_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc24236063c4c7aa912060d76892563"
I20260620 07:09:31.160888 8548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8b5877cfaa4799896c2f106d238c9e" candidate_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c65855a594b4570b8139916901d4181"
I20260620 07:09:31.161087 8548 raft_consensus.cc:3037] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:31.161131 8406 raft_consensus.cc:3037] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:31.163777 8406 raft_consensus.cc:2454] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb51a1e996ce4d68b6f3f3c6791d266f in term 1.
I20260620 07:09:31.164060 8548 raft_consensus.cc:2454] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb51a1e996ce4d68b6f3f3c6791d266f in term 1.
I20260620 07:09:31.164192 8197 leader_election.cc:302] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dc24236063c4c7aa912060d76892563, fb51a1e996ce4d68b6f3f3c6791d266f; no voters:
I20260620 07:09:31.164601 8620 raft_consensus.cc:2781] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:31.164762 8620 raft_consensus.cc:686] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 LEADER]: Becoming Leader. State: Replica: fb51a1e996ce4d68b6f3f3c6791d266f, State: Running, Role: LEADER
I20260620 07:09:31.165033 8620 consensus_queue.cc:237] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } }
I20260620 07:09:31.167788 8107 catalog_manager.cc:5495] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f reported cstate change: term changed from 0 to 1, leader changed from <none> to fb51a1e996ce4d68b6f3f3c6791d266f (127.6.53.65). New cstate: current_term: 1 leader_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb51a1e996ce4d68b6f3f3c6791d266f" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 } health_report { overall_health: UNKNOWN } } }
I20260620 07:09:31.189040 8709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:31.184108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54124 (local address 127.6.53.65:38753)
0620 07:09:31.184525 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:31.184532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:31.184553 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:09:31.184724 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:31.184730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:31.184802 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:31.184894 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:31.185262 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:31.186073 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:31.187365 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:31.187779 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:31.187923 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:31.188347 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:31.188450 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:31.188545 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:31.188673 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 07:09:31.188751 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295}
I20260620 07:09:31.201289 8709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:31.196601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:33697 (local address 127.6.53.65:38753)
0620 07:09:31.196736 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:31.196740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:31.196809 (+ 69us) server_negotiation.cc:407] Connection header received
0620 07:09:31.196933 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:31.196936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:31.197007 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:31.197100 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:31.197461 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:31.198243 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:31.199626 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:31.200048 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:31.200174 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:31.200590 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:31.200670 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:31.200756 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:31.200909 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 07:09:31.200974 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":42}
I20260620 07:09:31.201320 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:31.196479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38753 (local address 127.6.53.65:33697)
0620 07:09:31.196714 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:31.196734 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:31.196844 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:31.197120 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:31.197124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:31.197134 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:09:31.197341 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:31.197348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:31.198407 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:31.198412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:31.199500 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:31.199511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:31.200182 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:31.200187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:31.200392 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:31.200437 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:09:31.200766 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:09:31.200777 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 07:09:31.200864 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 07:09:31.201007 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":76}
I20260620 07:09:31.210291 8548 raft_consensus.cc:1261] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181 [term 1 FOLLOWER]: Refusing update from remote peer fb51a1e996ce4d68b6f3f3c6791d266f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:31.210305 8406 raft_consensus.cc:1261] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Refusing update from remote peer fb51a1e996ce4d68b6f3f3c6791d266f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:31.210868 8620 consensus_queue.cc:1035] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c65855a594b4570b8139916901d4181" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 38981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:31.211164 8676 consensus_queue.cc:1035] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dc24236063c4c7aa912060d76892563" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:31.227175 8673 mvcc.cc:205] Tried to move back new op lower bound from 7298823664473153536 to 7298823664294600704. Current Snapshot: MvccSnapshot[applied={T|T < 7298823664473153536}]
W20260620 07:09:31.239413 8669 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 07:09:32.826365 8105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:09:32.826624 8105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:09:32.830240 8105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7b757f9a3ff4c008ccdb7624c15b4f9: Sending DeleteTablet for 3 replicas of tablet 8b8b5877cfaa4799896c2f106d238c9e
I20260620 07:09:32.830955 8244 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b8b5877cfaa4799896c2f106d238c9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:32 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:54110
I20260620 07:09:32.831180 8383 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b8b5877cfaa4799896c2f106d238c9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:32 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:49238
I20260620 07:09:32.831296 8526 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b8b5877cfaa4799896c2f106d238c9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:32 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:42754
I20260620 07:09:32.831828 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 8178
I20260620 07:09:32.843575 8760 tablet_replica.cc:330] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181: stopping tablet replica
I20260620 07:09:32.843878 8760 raft_consensus.cc:2229] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:09:32.844177 8760 raft_consensus.cc:2258] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:32.844837 8758 tablet_replica.cc:330] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f: stopping tablet replica
I20260620 07:09:32.845046 8758 raft_consensus.cc:2229] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:09:32.845067 8759 tablet_replica.cc:330] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563: stopping tablet replica
I20260620 07:09:32.845268 8759 raft_consensus.cc:2229] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:09:32.845496 8759 raft_consensus.cc:2258] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:32.845634 8758 raft_consensus.cc:2258] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:32.845830 8760 ts_tablet_manager.cc:1905] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:32.846673 8758 ts_tablet_manager.cc:1905] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:32.846838 8759 ts_tablet_manager.cc:1905] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:32.852474 8760 ts_tablet_manager.cc:1918] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:09:32.852629 8760 log.cc:1196] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/8b8b5877cfaa4799896c2f106d238c9e
I20260620 07:09:32.853070 8760 ts_tablet_manager.cc:1939] T 8b8b5877cfaa4799896c2f106d238c9e P 2c65855a594b4570b8139916901d4181: Deleting consensus metadata
I20260620 07:09:32.853400 8758 ts_tablet_manager.cc:1918] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:09:32.853540 8758 log.cc:1196] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/8b8b5877cfaa4799896c2f106d238c9e
I20260620 07:09:32.853679 8759 ts_tablet_manager.cc:1918] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:09:32.853808 8759 log.cc:1196] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/8b8b5877cfaa4799896c2f106d238c9e
I20260620 07:09:32.854005 8758 ts_tablet_manager.cc:1939] T 8b8b5877cfaa4799896c2f106d238c9e P fb51a1e996ce4d68b6f3f3c6791d266f: Deleting consensus metadata
I20260620 07:09:32.854331 8759 ts_tablet_manager.cc:1939] T 8b8b5877cfaa4799896c2f106d238c9e P 9dc24236063c4c7aa912060d76892563: Deleting consensus metadata
I20260620 07:09:32.854302 8091 catalog_manager.cc:4841] TS 2c65855a594b4570b8139916901d4181 (127.6.53.67:38981): tablet 8b8b5877cfaa4799896c2f106d238c9e (table test-table [id=6f5295f3688944b185043e53d9834b4b]) successfully deleted
I20260620 07:09:32.855278 8094 catalog_manager.cc:4841] TS 9dc24236063c4c7aa912060d76892563 (127.6.53.66:42645): tablet 8b8b5877cfaa4799896c2f106d238c9e (table test-table [id=6f5295f3688944b185043e53d9834b4b]) successfully deleted
I20260620 07:09:32.855327 8093 catalog_manager.cc:4841] TS fb51a1e996ce4d68b6f3f3c6791d266f (127.6.53.65:38753): tablet 8b8b5877cfaa4799896c2f106d238c9e (table test-table [id=6f5295f3688944b185043e53d9834b4b]) successfully deleted
Jun 20 07:09:32 dist-test-slave-4n6h krb5kdc[8062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939370, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:09:32.861502 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:32.834320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.6.53.65:38753)
0620 07:09:32.839744 (+ 5424us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:32.839753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:32.843891 (+ 4138us) server_negotiation.cc:407] Connection header received
0620 07:09:32.844113 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:32.844118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:32.844192 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:32.844308 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:32.846387 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:32.847480 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:32.849323 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:32.850026 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:32.850140 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:32.856305 (+ 6165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:32.856318 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:32.856335 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:32.856370 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:32.859098 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:32.859693 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:32.859700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:32.859703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:32.859817 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:32.860144 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:32.860147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:32.860149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:32.860518 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:32.860723 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:32.860980 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 07:09:32.861234 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5246,"thread_start_us":5110,"threads_started":1}
W20260620 07:09:32.862179 8309 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57358
W20260620 07:09:32.862829 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 8178: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:09:32.863021 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 8178
I20260620 07:09:32.889084 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 8318
Jun 20 07:09:32 dist-test-slave-4n6h krb5kdc[8062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939370, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:09:32.903594 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:32.889742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47296 (local address 127.6.53.66:42645)
0620 07:09:32.890329 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:32.890333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:32.890349 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:09:32.890398 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:32.890401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:32.890457 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:32.890546 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:32.891715 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:32.892476 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:32.893662 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:32.894079 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:32.894181 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:32.898424 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:32.898438 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:32.898453 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:32.898492 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:32.900964 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:32.901641 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:32.901648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:32.901651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:32.901795 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:32.902118 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:32.902121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:32.902123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:32.902594 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:32.902844 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:32.903075 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 07:09:32.903269 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":329,"threads_started":1}
W20260620 07:09:32.904269 8448 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47296
W20260620 07:09:32.904824 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 8318: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:09:32.904983 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 8318
I20260620 07:09:32.925993 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 8457
Jun 20 07:09:32 dist-test-slave-4n6h krb5kdc[8062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939370, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:09:32.941152 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:32.926853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.6.53.67:38981)
0620 07:09:32.927463 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:32.927468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:32.927486 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:09:32.927542 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:32.927546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:32.927607 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:32.927707 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:32.928921 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:32.929689 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:32.930915 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:32.931320 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:32.931411 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:32.935580 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:32.935594 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:32.935611 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:32.935647 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:32.938627 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:32.939243 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:32.939250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:32.939252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:32.939437 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:32.939788 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:32.939791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:32.939793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:32.940147 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:32.940433 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:32.940720 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 07:09:32.940874 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":347,"threads_started":1}
W20260620 07:09:32.941843 8588 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46746
W20260620 07:09:32.942466 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 8457: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:09:32.942646 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 8457
I20260620 07:09:32.964479 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 8075
I20260620 07:09:32.965716 8147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:33.079974 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 8075
2026-06-20T07:09:33Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8340 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:09:33 dist-test-slave-4n6h krb5kdc[8768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:09:33 dist-test-slave-4n6h krb5kdc[8768](info): set up 2 sockets
Jun 20 07:09:33 dist-test-slave-4n6h krb5kdc[8768](info): commencing operation
krb5kdc: starting...
W20260620 07:09:34.186553 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:09:34 dist-test-slave-4n6h krb5kdc[8768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:09:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:09:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:09:34.369536 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:35395
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:35395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:34.687783 8781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:34.688063 8781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:34.688156 8781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:34.695228 8781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:34.695319 8781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:34.695367 8781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:34.695423 8781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:34.695473 8781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:34.704926 8781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:35395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:35395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:34.706090 8781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:34.707320 8781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:34.718762 8786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:34.718740 8787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:34.718935 8789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:34.719956 8781 server_base.cc:1034] running on GCE node
I20260620 07:09:34.720741 8781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:34.722056 8781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:34.723282 8781 hybrid_clock.cc:648] HybridClock initialized: now 1781939374723252 us; error 58 us; skew 500 ppm
I20260620 07:09:34.723621 8781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:34 dist-test-slave-4n6h krb5kdc[8768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939374, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:34.726552 8781 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:09:34.728026 8781 webserver.cc:460] Webserver started at http://127.6.53.126:43331/ using document root <none> and password file <none>
I20260620 07:09:34.728679 8781 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:34.728819 8781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:34.729122 8781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:34.732017 8781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "e432d11c73fd423583c98850c52e98ea"
format_stamp: "Formatted at 2026-06-20 07:09:34 on dist-test-slave-4n6h"
server_key: "cc69743109e214aff175b50ca266df34"
server_key_iv: "eaf358b67acb8df91bb02fa9697ed97d"
server_key_version: "encryptionkey@0"
I20260620 07:09:34.732772 8781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "e432d11c73fd423583c98850c52e98ea"
format_stamp: "Formatted at 2026-06-20 07:09:34 on dist-test-slave-4n6h"
server_key: "cc69743109e214aff175b50ca266df34"
server_key_iv: "eaf358b67acb8df91bb02fa9697ed97d"
server_key_version: "encryptionkey@0"
I20260620 07:09:34.737687 8781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:09:34.740911 8796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:34.742152 8781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:09:34.742348 8781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "e432d11c73fd423583c98850c52e98ea"
format_stamp: "Formatted at 2026-06-20 07:09:34 on dist-test-slave-4n6h"
server_key: "cc69743109e214aff175b50ca266df34"
server_key_iv: "eaf358b67acb8df91bb02fa9697ed97d"
server_key_version: "encryptionkey@0"
I20260620 07:09:34.742542 8781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:34.777161 8781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:34.781085 8781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:34.781368 8781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:34.799888 8781 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:35395
I20260620 07:09:34.799923 8847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:35395 every 8 connection(s)
I20260620 07:09:34.801235 8781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:34.804852 8848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:34.806314 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 8781
I20260620 07:09:34.806530 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:09:34.807169 6357 external_mini_cluster.cc:1428] Setting key e6435e1b23c83e85db5f9f26884cf51e
I20260620 07:09:34.818881 8848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea: Bootstrap starting.
Jun 20 07:09:34 dist-test-slave-4n6h krb5kdc[8768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:34.821810 8848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:34.823050 8848 log.cc:824] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:34.826416 8848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea: No bootstrap required, opened a new log
I20260620 07:09:34.830821 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:34.809229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40208 (local address 127.6.53.126:35395)
0620 07:09:34.810071 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:34.810086 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:34.810132 (+ 46us) server_negotiation.cc:407] Connection header received
0620 07:09:34.811921 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:34.811949 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:34.812467 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:34.812862 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:34.814125 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:34.816194 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:34.817456 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:34.818479 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:34.818665 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:34.822416 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:34.822427 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:34.822444 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:34.822489 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:34.826305 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:34.827088 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:34.827098 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:34.827106 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:34.827309 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:34.827673 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:34.827678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:34.827681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:34.828350 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:34.828627 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:34.828999 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 07:09:34.829436 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":367,"threads_started":1}
I20260620 07:09:34.835875 8848 raft_consensus.cc:348] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e432d11c73fd423583c98850c52e98ea" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35395 } }
I20260620 07:09:34.836246 8848 raft_consensus.cc:374] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:34.836349 8848 raft_consensus.cc:729] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e432d11c73fd423583c98850c52e98ea, State: Initialized, Role: FOLLOWER
I20260620 07:09:34.836970 8848 consensus_queue.cc:260] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e432d11c73fd423583c98850c52e98ea" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35395 } }
I20260620 07:09:34.837245 8848 raft_consensus.cc:388] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:09:34.837390 8848 raft_consensus.cc:482] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:09:34.837572 8848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:34.839721 8848 raft_consensus.cc:504] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e432d11c73fd423583c98850c52e98ea" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35395 } }
I20260620 07:09:34.840219 8848 leader_election.cc:302] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e432d11c73fd423583c98850c52e98ea; no voters:
I20260620 07:09:34.840807 8848 leader_election.cc:288] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:09:34.840950 8853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:34.842569 8853 raft_consensus.cc:686] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [term 1 LEADER]: Becoming Leader. State: Replica: e432d11c73fd423583c98850c52e98ea, State: Running, Role: LEADER
I20260620 07:09:34.842984 8848 sys_catalog.cc:564] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:34.843116 8853 consensus_queue.cc:237] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e432d11c73fd423583c98850c52e98ea" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35395 } }
I20260620 07:09:34.845703 8855 sys_catalog.cc:455] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader e432d11c73fd423583c98850c52e98ea. Latest consensus state: current_term: 1 leader_uuid: "e432d11c73fd423583c98850c52e98ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e432d11c73fd423583c98850c52e98ea" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35395 } } }
I20260620 07:09:34.846127 8854 sys_catalog.cc:455] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e432d11c73fd423583c98850c52e98ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e432d11c73fd423583c98850c52e98ea" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35395 } } }
I20260620 07:09:34.846415 8855 sys_catalog.cc:458] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:34.846449 8854 sys_catalog.cc:458] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:34.851296 8862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:09:34.856406 8862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:09:34.865448 8862 catalog_manager.cc:1347] Generated new cluster ID: 28af52ca7fcc46c7985ee21bb58ff4ab
I20260620 07:09:34.865595 8862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:34.876650 8862 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:34.878667 8862 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:34.885033 8862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e432d11c73fd423583c98850c52e98ea: Generated new TSK 0
I20260620 07:09:34.885950 8862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:09:34.930821 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:35395
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:35.250130 8874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:35.250463 8874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:35.250644 8874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:35.257737 8874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:35.257838 8874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:35.258018 8874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:09:35.267572 8874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:35395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:35.268790 8874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:35.270083 8874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:35.283420 8882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:35.283458 8880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:35.284264 8879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:35.288106 8874 server_base.cc:1034] running on GCE node
I20260620 07:09:35.288774 8874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:35.289748 8874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:35.290940 8874 hybrid_clock.cc:648] HybridClock initialized: now 1781939375290889 us; error 40 us; skew 500 ppm
I20260620 07:09:35.291268 8874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:35 dist-test-slave-4n6h krb5kdc[8768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939375, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:35.294471 8874 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:09:35.295953 8874 webserver.cc:460] Webserver started at http://127.6.53.65:33833/ using document root <none> and password file <none>
I20260620 07:09:35.296671 8874 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:35.296801 8874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:35.297119 8874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:35.300014 8874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "343705cf69514fd89f85398a07ede7e1"
format_stamp: "Formatted at 2026-06-20 07:09:35 on dist-test-slave-4n6h"
server_key: "d6a163dd1b6cdc88aa16c5dafc3fd798"
server_key_iv: "8474eb239de881cf12a09d5464467f0d"
server_key_version: "encryptionkey@0"
I20260620 07:09:35.300773 8874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "343705cf69514fd89f85398a07ede7e1"
format_stamp: "Formatted at 2026-06-20 07:09:35 on dist-test-slave-4n6h"
server_key: "d6a163dd1b6cdc88aa16c5dafc3fd798"
server_key_iv: "8474eb239de881cf12a09d5464467f0d"
server_key_version: "encryptionkey@0"
I20260620 07:09:35.305631 8874 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:09:35.308944 8889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:35.310146 8874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 07:09:35.310345 8874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "343705cf69514fd89f85398a07ede7e1"
format_stamp: "Formatted at 2026-06-20 07:09:35 on dist-test-slave-4n6h"
server_key: "d6a163dd1b6cdc88aa16c5dafc3fd798"
server_key_iv: "8474eb239de881cf12a09d5464467f0d"
server_key_version: "encryptionkey@0"
I20260620 07:09:35.310542 8874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:35.321552 8874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:35.325729 8874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:35.326010 8874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:35.326969 8874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:35.328241 8874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:35.328341 8874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:35.328485 8874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:35.328557 8874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:35.362129 8874 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:33207
I20260620 07:09:35.362195 9001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:33207 every 8 connection(s)
I20260620 07:09:35.363425 8874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:09:35.367772 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 8874
I20260620 07:09:35.367995 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:09:35.368626 6357 external_mini_cluster.cc:1428] Setting key fc8b49f73146f6a2803ceff0d615fdb2
Jun 20 07:09:35 dist-test-slave-4n6h krb5kdc[8768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939375, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:35.384538 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:35.365874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:39989 (local address 127.6.53.126:35395)
0620 07:09:35.366346 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:35.366351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:35.367192 (+ 841us) server_negotiation.cc:407] Connection header received
0620 07:09:35.369106 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:35.369112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:35.369202 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:35.369320 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:35.370827 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:35.371521 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:35.373036 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:35.373439 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:35.373582 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:35.377529 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:35.377533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:35.377537 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:35.377568 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:35.381056 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:35.381890 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:35.381898 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:35.381904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:35.382058 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:35.382600 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:35.382606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:35.382609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:35.382995 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:35.383160 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:35.383814 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 07:09:35.384090 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":228,"threads_started":1}
I20260620 07:09:35.385370 9005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:35.366355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:35395 (local address 127.6.53.65:39989)
0620 07:09:35.367030 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:35.367081 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:35.368854 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:35.369545 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:35.369554 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:35.370049 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:35.370667 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:35.370684 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:35.371678 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:35.371687 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:35.372893 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:35.372908 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:35.373652 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:35.373660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:35.373892 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:35.374650 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:35.374686 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:35.377308 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:35.381288 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:35.381294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:35.381306 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:35.381750 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:35.382174 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:35.382177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:35.382180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:35.382472 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:35.383126 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:35.383132 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:35.383428 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 07:09:35.383657 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 07:09:35.384028 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":267,"threads_started":1}
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
I20260620 07:09:35.387022 9002 heartbeater.cc:347] Connected to a master server at 127.6.53.126:35395
I20260620 07:09:35.387315 9002 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:35.388084 9002 heartbeater.cc:511] Master 127.6.53.126:35395 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
I20260620 07:09:35.390218 8813 ts_manager.cc:194] Registered new tserver with Master: 343705cf69514fd89f85398a07ede7e1 (127.6.53.65:33207)
I20260620 07:09:35.391709 8813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:39989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:09:35.409612 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:35395
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:35.720340 9008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:35.720631 9008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:35.720824 9008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:35.728235 9008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:35.728370 9008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:35.728578 9008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:09:35.737861 9008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:35395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:35.739094 9008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:35.740340 9008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:35.752995 9013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:35.753072 9014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:35.753028 9016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:35.753859 9008 server_base.cc:1034] running on GCE node
I20260620 07:09:35.754525 9008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:35.755414 9008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:35.756642 9008 hybrid_clock.cc:648] HybridClock initialized: now 1781939375756595 us; error 57 us; skew 500 ppm
I20260620 07:09:35.756943 9008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:35 dist-test-slave-4n6h krb5kdc[8768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939375, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:35.760059 9008 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:09:35.761458 9008 webserver.cc:460] Webserver started at http://127.6.53.66:38115/ using document root <none> and password file <none>
I20260620 07:09:35.762086 9008 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:35.762220 9008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:35.762539 9008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:35.765305 9008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "2d91fc9ba0d347f2b9c9a1655433ad3a"
format_stamp: "Formatted at 2026-06-20 07:09:35 on dist-test-slave-4n6h"
server_key: "1ba4eae6406eed6739863aaf698c65c8"
server_key_iv: "35dc21b401c19a66d0223594151e009e"
server_key_version: "encryptionkey@0"
I20260620 07:09:35.766052 9008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "2d91fc9ba0d347f2b9c9a1655433ad3a"
format_stamp: "Formatted at 2026-06-20 07:09:35 on dist-test-slave-4n6h"
server_key: "1ba4eae6406eed6739863aaf698c65c8"
server_key_iv: "35dc21b401c19a66d0223594151e009e"
server_key_version: "encryptionkey@0"
I20260620 07:09:35.770800 9008 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260620 07:09:35.773965 9023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:35.775269 9008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:35.775439 9008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "2d91fc9ba0d347f2b9c9a1655433ad3a"
format_stamp: "Formatted at 2026-06-20 07:09:35 on dist-test-slave-4n6h"
server_key: "1ba4eae6406eed6739863aaf698c65c8"
server_key_iv: "35dc21b401c19a66d0223594151e009e"
server_key_version: "encryptionkey@0"
I20260620 07:09:35.775646 9008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:35.805663 9008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:35.810716 9008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:35.810995 9008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:35.812031 9008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:35.813575 9008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:35.813660 9008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:35.813767 9008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:35.813822 9008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:35.854970 9008 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:37351
I20260620 07:09:35.855018 9135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:37351 every 8 connection(s)
I20260620 07:09:35.856137 9008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:09:35.856416 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 9008
I20260620 07:09:35.856666 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:09:35.857225 6357 external_mini_cluster.cc:1428] Setting key 318ec0cc6a44c74d13ac108543a64fe2
Jun 20 07:09:35 dist-test-slave-4n6h krb5kdc[8768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939375, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
I20260620 07:09:35.876479 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:35.858964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:45363 (local address 127.6.53.126:35395)
0620 07:09:35.859424 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:35.859430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:35.860409 (+ 979us) server_negotiation.cc:407] Connection header received
0620 07:09:35.862186 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:35.862195 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:35.862309 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:35.862412 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:35.863861 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:35.864570 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:35.866174 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:35.866667 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:35.866802 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:35.870182 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:35.870187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:35.870190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:35.870220 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:35.873178 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:35.874025 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:35.874031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:35.874041 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:35.874186 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:35.874692 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:35.874699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:35.874703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:35.874994 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:35.875154 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:35.876041 (+ 887us) server_negotiation.cc:299] Negotiation successful
0620 07:09:35.876249 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324}
I20260620 07:09:35.878000 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:35.859436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:35395 (local address 127.6.53.66:45363)
0620 07:09:35.860225 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:35.860276 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:35.861928 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:35.862625 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:35.862632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:35.863149 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:35.863696 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:35.863714 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:35.864738 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:35.864743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:35.866023 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:35.866035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:35.866889 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:35.866896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:35.867121 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:35.867800 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:35.867822 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:35.869959 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:35.873339 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:35.873349 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:35.873367 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:35.873875 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:35.874350 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:35.874355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:35.874360 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:35.874567 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:35.875140 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:35.875148 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:35.875506 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 07:09:35.875805 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 07:09:35.876378 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":343,"threads_started":1}
I20260620 07:09:35.879673 9136 heartbeater.cc:347] Connected to a master server at 127.6.53.126:35395
I20260620 07:09:35.880023 9136 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:35.880716 9136 heartbeater.cc:511] Master 127.6.53.126:35395 requested a full tablet report, sending...
I20260620 07:09:35.882136 8813 ts_manager.cc:194] Registered new tserver with Master: 2d91fc9ba0d347f2b9c9a1655433ad3a (127.6.53.66:37351)
I20260620 07:09:35.883011 8813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:45363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:09:35.899587 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:35395
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:36.217343 9141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:36.217597 9141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:36.217760 9141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:36.224853 9141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:36.224999 9141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:36.225140 9141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:09:36.234359 9141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:35395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:36.235592 9141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:36.236807 9141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:36.249320 9147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:36.249406 9146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:36.249351 9149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:36.250524 9141 server_base.cc:1034] running on GCE node
I20260620 07:09:36.251030 9141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:36.251874 9141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:36.253099 9141 hybrid_clock.cc:648] HybridClock initialized: now 1781939376253065 us; error 46 us; skew 500 ppm
I20260620 07:09:36.253412 9141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:36 dist-test-slave-4n6h krb5kdc[8768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939376, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:36.256413 9141 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:09:36.257879 9141 webserver.cc:460] Webserver started at http://127.6.53.67:43553/ using document root <none> and password file <none>
I20260620 07:09:36.258620 9141 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:36.258762 9141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:36.259068 9141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:36.261829 9141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "b506f2fb74ed4873b2633d1e2588b8cb"
format_stamp: "Formatted at 2026-06-20 07:09:36 on dist-test-slave-4n6h"
server_key: "31f5523b4bc7d7798cb93faeedf3980c"
server_key_iv: "4f66ddeccab0e116dd2581de230a1cfe"
server_key_version: "encryptionkey@0"
I20260620 07:09:36.262655 9141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "b506f2fb74ed4873b2633d1e2588b8cb"
format_stamp: "Formatted at 2026-06-20 07:09:36 on dist-test-slave-4n6h"
server_key: "31f5523b4bc7d7798cb93faeedf3980c"
server_key_iv: "4f66ddeccab0e116dd2581de230a1cfe"
server_key_version: "encryptionkey@0"
I20260620 07:09:36.267443 9141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 07:09:36.270715 9156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:36.271895 9141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:09:36.272060 9141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "b506f2fb74ed4873b2633d1e2588b8cb"
format_stamp: "Formatted at 2026-06-20 07:09:36 on dist-test-slave-4n6h"
server_key: "31f5523b4bc7d7798cb93faeedf3980c"
server_key_iv: "4f66ddeccab0e116dd2581de230a1cfe"
server_key_version: "encryptionkey@0"
I20260620 07:09:36.272253 9141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:36.296977 9141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:36.302501 9141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:36.302801 9141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:36.303718 9141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:36.304979 9141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:36.305085 9141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:36.305231 9141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:36.305302 9141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:36.335915 9141 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:34683
I20260620 07:09:36.335966 9268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:34683 every 8 connection(s)
I20260620 07:09:36.337150 9141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:09:36.347150 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 9141
I20260620 07:09:36.347333 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:09:36.347764 6357 external_mini_cluster.cc:1428] Setting key 1bdf781161edfd53a6931584c7d9b226
Jun 20 07:09:36 dist-test-slave-4n6h krb5kdc[8768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939376, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:36.357168 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:36.339764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:53897 (local address 127.6.53.126:35395)
0620 07:09:36.339955 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:36.339960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:36.341031 (+ 1071us) server_negotiation.cc:407] Connection header received
0620 07:09:36.342946 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:36.342953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:36.343049 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:36.343160 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:36.344644 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:36.345419 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:36.347080 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:36.347617 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:36.347747 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:36.351678 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:36.351683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:36.351686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:36.351718 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:36.354227 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:36.354965 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:36.354970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:36.354974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:36.355105 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:36.355526 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:36.355531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:36.355534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:36.355804 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:36.355955 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:36.356609 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 07:09:36.356834 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 07:09:36.358187 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:36.340189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:35395 (local address 127.6.53.67:53897)
0620 07:09:36.340871 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:36.340912 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:36.342657 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:36.343382 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:36.343390 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:36.343953 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:36.344479 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:36.344495 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:36.345575 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:36.345579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:36.346945 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:36.346956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:36.347845 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:36.347853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:36.348081 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:36.348773 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:36.348794 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:36.351473 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:36.354407 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:36.354416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:36.354431 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:36.354843 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:36.355218 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:36.355223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:36.355227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:36.355412 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:36.355929 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:36.355936 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:36.356264 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 07:09:36.356516 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 07:09:36.357019 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":264,"threads_started":1}
I20260620 07:09:36.359689 9269 heartbeater.cc:347] Connected to a master server at 127.6.53.126:35395
I20260620 07:09:36.359984 9269 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:36.360677 9269 heartbeater.cc:511] Master 127.6.53.126:35395 requested a full tablet report, sending...
I20260620 07:09:36.362334 8812 ts_manager.cc:194] Registered new tserver with Master: b506f2fb74ed4873b2633d1e2588b8cb (127.6.53.67:34683)
I20260620 07:09:36.363476 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:09:36.363464 8812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:53897
W20260620 07:09:36.393559 9004 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:36.391070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.6.53.126:35395)
0620 07:09:36.391327 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:36.391335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:36.391639 (+ 304us) server_negotiation.cc:407] Connection header received
0620 07:09:36.391857 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:36.391861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:36.391923 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:36.392033 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:36.393276 (+ 1243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40224: BlockingRecv error: recv got EOF from 127.0.0.1:40224 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 07:09:36.394515 9002 heartbeater.cc:503] Master 127.6.53.126:35395 was elected leader, sending a full tablet report...
I20260620 07:09:36.397533 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 8874
W20260620 07:09:36.400324 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 8874: Not authorized: Client connection negotiation failed: client connection to 127.6.53.65:33207: server requires authentication, but client does not have Kerberos credentials available
I20260620 07:09:36.400507 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 8874
I20260620 07:09:36.415997 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 9008
W20260620 07:09:36.418669 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 9008: Not authorized: Client connection negotiation failed: client connection to 127.6.53.66:37351: server requires authentication, but client does not have Kerberos credentials available
I20260620 07:09:36.418841 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 9008
I20260620 07:09:36.433915 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 9141
W20260620 07:09:36.436741 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 9141: Not authorized: Client connection negotiation failed: client connection to 127.6.53.67:34683: server requires authentication, but client does not have Kerberos credentials available
I20260620 07:09:36.436883 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 9141
I20260620 07:09:36.452318 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 8781
I20260620 07:09:36.453467 8843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:36.553191 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 8781
2026-06-20T07:09:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3470 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:09:36 dist-test-slave-4n6h krb5kdc[9286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:09:36 dist-test-slave-4n6h krb5kdc[9286](info): set up 2 sockets
Jun 20 07:09:36 dist-test-slave-4n6h krb5kdc[9286](info): commencing operation
krb5kdc: starting...
W20260620 07:09:37.654583 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:09:37 dist-test-slave-4n6h krb5kdc[9286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:09:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:09:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:09:37.833523 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:45613
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:45613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:38.147195 9299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:38.147480 9299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:38.147586 9299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:38.154706 9299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:38.154814 9299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:38.154865 9299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:38.154929 9299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:38.154983 9299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:38.164928 9299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:45613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:45613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:38.166232 9299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:38.167573 9299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:38.179559 9305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:38.180295 9307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:38.180431 9304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:38.180738 9299 server_base.cc:1034] running on GCE node
I20260620 07:09:38.181564 9299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:38.183120 9299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:38.184357 9299 hybrid_clock.cc:648] HybridClock initialized: now 1781939378184310 us; error 41 us; skew 500 ppm
I20260620 07:09:38.184729 9299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:38 dist-test-slave-4n6h krb5kdc[9286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939378, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:38.192368 9299 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:09:38.193889 9299 webserver.cc:460] Webserver started at http://127.6.53.126:41819/ using document root <none> and password file <none>
I20260620 07:09:38.194640 9299 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:38.194778 9299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:38.195081 9299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:38.197993 9299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "a6d6aafa66ed45cdbbcf4b452b95317b"
format_stamp: "Formatted at 2026-06-20 07:09:38 on dist-test-slave-4n6h"
server_key: "96fdc75bb5eb2033a3cef1b56e2e45d9"
server_key_iv: "ffc3a3571b61b1b55d758041eebba2f5"
server_key_version: "encryptionkey@0"
I20260620 07:09:38.198837 9299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "a6d6aafa66ed45cdbbcf4b452b95317b"
format_stamp: "Formatted at 2026-06-20 07:09:38 on dist-test-slave-4n6h"
server_key: "96fdc75bb5eb2033a3cef1b56e2e45d9"
server_key_iv: "ffc3a3571b61b1b55d758041eebba2f5"
server_key_version: "encryptionkey@0"
I20260620 07:09:38.203881 9299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 07:09:38.207170 9314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:38.208428 9299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:09:38.208602 9299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "a6d6aafa66ed45cdbbcf4b452b95317b"
format_stamp: "Formatted at 2026-06-20 07:09:38 on dist-test-slave-4n6h"
server_key: "96fdc75bb5eb2033a3cef1b56e2e45d9"
server_key_iv: "ffc3a3571b61b1b55d758041eebba2f5"
server_key_version: "encryptionkey@0"
I20260620 07:09:38.208802 9299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:38.234313 9299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:38.238174 9299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:38.238494 9299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:38.256385 9299 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:45613
I20260620 07:09:38.256422 9365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:45613 every 8 connection(s)
I20260620 07:09:38.257681 9299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:38.261193 9366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:38.261807 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 9299
I20260620 07:09:38.261999 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:09:38.262825 6357 external_mini_cluster.cc:1428] Setting key bcd7ed719fc10a1989e4db9f44046ff3
I20260620 07:09:38.274911 9366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b: Bootstrap starting.
Jun 20 07:09:38 dist-test-slave-4n6h krb5kdc[9286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:38.278395 9366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:38.279552 9366 log.cc:824] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:38.282760 9366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b: No bootstrap required, opened a new log
I20260620 07:09:38.286221 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:38.264734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.6.53.126:45613)
0620 07:09:38.265515 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:38.265526 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:38.265571 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:09:38.267427 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:38.267448 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:38.267837 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:38.268150 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:38.269869 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:38.271168 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:38.272987 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:38.273600 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:38.273750 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:38.278227 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:38.278238 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:38.278312 (+ 74us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:38.278352 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:38.281871 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:38.282620 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:38.282628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:38.282635 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:38.282855 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:38.283220 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:38.283224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:38.283227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:38.283921 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:38.284194 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:38.284475 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 07:09:38.284924 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":367,"threads_started":1}
I20260620 07:09:38.289983 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:09:38.291996 9366 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d6aafa66ed45cdbbcf4b452b95317b" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45613 } }
I20260620 07:09:38.292313 9366 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:38.292421 9366 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d6aafa66ed45cdbbcf4b452b95317b, State: Initialized, Role: FOLLOWER
I20260620 07:09:38.292995 9366 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d6aafa66ed45cdbbcf4b452b95317b" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45613 } }
I20260620 07:09:38.293218 9366 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:09:38.293335 9366 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:09:38.293506 9366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:38.295606 9366 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d6aafa66ed45cdbbcf4b452b95317b" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45613 } }
I20260620 07:09:38.296181 9366 leader_election.cc:302] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6d6aafa66ed45cdbbcf4b452b95317b; no voters:
I20260620 07:09:38.296792 9366 leader_election.cc:288] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:09:38.296864 9372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:38.298179 9372 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [term 1 LEADER]: Becoming Leader. State: Replica: a6d6aafa66ed45cdbbcf4b452b95317b, State: Running, Role: LEADER
I20260620 07:09:38.298683 9372 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d6aafa66ed45cdbbcf4b452b95317b" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45613 } }
I20260620 07:09:38.298949 9366 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:38.301059 9374 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6d6aafa66ed45cdbbcf4b452b95317b. Latest consensus state: current_term: 1 leader_uuid: "a6d6aafa66ed45cdbbcf4b452b95317b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d6aafa66ed45cdbbcf4b452b95317b" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45613 } } }
I20260620 07:09:38.301394 9374 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:38.301892 9373 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6d6aafa66ed45cdbbcf4b452b95317b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6d6aafa66ed45cdbbcf4b452b95317b" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45613 } } }
I20260620 07:09:38.302206 9373 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:38.309701 9377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:09:38.314952 9377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:09:38.345229 9377 catalog_manager.cc:1347] Generated new cluster ID: 7e2722dd48b848de9b00b71ad47d543a
I20260620 07:09:38.345626 9377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:38.357097 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:38.348127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37026 (local address 127.6.53.126:45613)
0620 07:09:38.348287 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:38.348293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:38.348900 (+ 607us) server_negotiation.cc:407] Connection header received
0620 07:09:38.349066 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:38.349072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:38.349156 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:38.349284 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:38.350449 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:38.351600 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:38.353335 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:38.354064 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:38.354210 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:38.355411 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:38.355418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:38.355422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:38.356310 (+ 888us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:38.356371 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:38.356400 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:38.356516 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:38.356639 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 07:09:38.356741 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:09:38.380342 9377 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:38.381752 9377 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:38.393930 9377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6d6aafa66ed45cdbbcf4b452b95317b: Generated new TSK 0
I20260620 07:09:38.394848 9377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:09:38.405283 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 9299
I20260620 07:09:38.406713 9361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:38.497890 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 9299
2026-06-20T07:09:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1938 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:09:38 dist-test-slave-4n6h krb5kdc[9401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:09:38 dist-test-slave-4n6h krb5kdc[9401](info): set up 2 sockets
Jun 20 07:09:38 dist-test-slave-4n6h krb5kdc[9401](info): commencing operation
krb5kdc: starting...
W20260620 07:09:39.574559 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:09:39 dist-test-slave-4n6h krb5kdc[9401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:09:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:09:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:09:39.754814 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:46345
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:46345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:40.063442 9414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:40.063705 9414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:40.063894 9414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:40.070844 9414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:40.070963 9414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:40.071064 9414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:40.071156 9414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:40.071228 9414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:40.080462 9414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:46345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:46345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:40.081677 9414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:40.082973 9414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:40.095152 9422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:40.095219 9420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:40.095314 9419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:40.096066 9414 server_base.cc:1034] running on GCE node
I20260620 07:09:40.096822 9414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:40.098037 9414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:40.099249 9414 hybrid_clock.cc:648] HybridClock initialized: now 1781939380099185 us; error 32 us; skew 500 ppm
I20260620 07:09:40.099659 9414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:40 dist-test-slave-4n6h krb5kdc[9401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939380, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:40.102735 9414 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:09:40.104071 9414 webserver.cc:460] Webserver started at http://127.6.53.126:33959/ using document root <none> and password file <none>
I20260620 07:09:40.104664 9414 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:40.104773 9414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:40.105010 9414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:40.107810 9414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "10024b7d32544cc88ba84149041455c9"
format_stamp: "Formatted at 2026-06-20 07:09:40 on dist-test-slave-4n6h"
server_key: "59c5d9c1ca2587a455afc12f81ac6fa3"
server_key_iv: "79b0ae922b99d724bfe09178233563b6"
server_key_version: "encryptionkey@0"
I20260620 07:09:40.108464 9414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "10024b7d32544cc88ba84149041455c9"
format_stamp: "Formatted at 2026-06-20 07:09:40 on dist-test-slave-4n6h"
server_key: "59c5d9c1ca2587a455afc12f81ac6fa3"
server_key_iv: "79b0ae922b99d724bfe09178233563b6"
server_key_version: "encryptionkey@0"
I20260620 07:09:40.113258 9414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:09:40.116406 9429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:40.117626 9414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 07:09:40.117755 9414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "10024b7d32544cc88ba84149041455c9"
format_stamp: "Formatted at 2026-06-20 07:09:40 on dist-test-slave-4n6h"
server_key: "59c5d9c1ca2587a455afc12f81ac6fa3"
server_key_iv: "79b0ae922b99d724bfe09178233563b6"
server_key_version: "encryptionkey@0"
I20260620 07:09:40.117897 9414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:40.136731 9414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:40.141340 9414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:40.141566 9414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:40.160034 9414 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:46345
I20260620 07:09:40.160056 9480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:46345 every 8 connection(s)
I20260620 07:09:40.161392 9414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:40.165010 9481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:40.170086 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 9414
I20260620 07:09:40.170361 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:09:40.170971 6357 external_mini_cluster.cc:1428] Setting key 73eff3ebe00fad8e7f85eb05ab864589
I20260620 07:09:40.175495 9481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9: Bootstrap starting.
I20260620 07:09:40.178996 9481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:40.180177 9481 log.cc:824] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:40.182870 9481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9: No bootstrap required, opened a new log
Jun 20 07:09:40 dist-test-slave-4n6h krb5kdc[9401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:40.190774 9481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10024b7d32544cc88ba84149041455c9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46345 } }
I20260620 07:09:40.191123 9481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:40.191236 9481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10024b7d32544cc88ba84149041455c9, State: Initialized, Role: FOLLOWER
I20260620 07:09:40.192021 9481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10024b7d32544cc88ba84149041455c9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46345 } }
I20260620 07:09:40.192315 9481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:09:40.192473 9481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:09:40.192687 9481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:40.193573 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:40.172856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.6.53.126:46345)
0620 07:09:40.173644 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:40.173655 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:40.173694 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:09:40.175350 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:40.175371 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:40.175777 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:40.176078 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:40.177840 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:40.179171 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:40.180921 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:40.181590 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:40.181771 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:40.185842 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:40.185850 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:40.185870 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:40.185918 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:40.189084 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:40.189763 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:40.189772 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:40.189787 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:40.189954 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:40.190346 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:40.190353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:40.190356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:40.190844 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:40.191172 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:40.191562 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 07:09:40.192107 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":386,"threads_started":1}
I20260620 07:09:40.195479 9481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10024b7d32544cc88ba84149041455c9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46345 } }
I20260620 07:09:40.196009 9481 leader_election.cc:302] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10024b7d32544cc88ba84149041455c9; no voters:
I20260620 07:09:40.196602 9481 leader_election.cc:288] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:09:40.196712 9486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:40.197422 9486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [term 1 LEADER]: Becoming Leader. State: Replica: 10024b7d32544cc88ba84149041455c9, State: Running, Role: LEADER
I20260620 07:09:40.197887 9486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10024b7d32544cc88ba84149041455c9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46345 } }
I20260620 07:09:40.198798 9481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:40.202519 9487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10024b7d32544cc88ba84149041455c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10024b7d32544cc88ba84149041455c9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46345 } } }
I20260620 07:09:40.202982 9487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:40.203574 9488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10024b7d32544cc88ba84149041455c9. Latest consensus state: current_term: 1 leader_uuid: "10024b7d32544cc88ba84149041455c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10024b7d32544cc88ba84149041455c9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46345 } } }
I20260620 07:09:40.203956 9488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:40.213512 9497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:09:40.217880 9497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:09:40.226563 9497 catalog_manager.cc:1347] Generated new cluster ID: fb8624cc99ec4550afc0fdd8f3ef1203
I20260620 07:09:40.226699 9497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:40.239895 9497 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:40.241573 9497 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:40.248450 9497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10024b7d32544cc88ba84149041455c9: Generated new TSK 0
I20260620 07:09:40.249190 9497 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:09:40.289656 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:46345
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:40.609591 9507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:40.609849 9507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:40.609999 9507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:40.617672 9507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:40.617794 9507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:40.617945 9507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:09:40.627672 9507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:46345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:40.628856 9507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:40.630118 9507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:40.642624 9512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:40.642757 9513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:40.642722 9515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:40.643893 9507 server_base.cc:1034] running on GCE node
I20260620 07:09:40.644423 9507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:40.645445 9507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:40.646627 9507 hybrid_clock.cc:648] HybridClock initialized: now 1781939380646570 us; error 49 us; skew 500 ppm
I20260620 07:09:40.646981 9507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:40 dist-test-slave-4n6h krb5kdc[9401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939380, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:40.650059 9507 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:09:40.651561 9507 webserver.cc:460] Webserver started at http://127.6.53.65:34481/ using document root <none> and password file <none>
I20260620 07:09:40.652225 9507 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:40.652351 9507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:40.652645 9507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:40.655483 9507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "6f77b1d76e304573a962e43efb94960e"
format_stamp: "Formatted at 2026-06-20 07:09:40 on dist-test-slave-4n6h"
server_key: "0bd0ee295421a708e68f20c9104a6724"
server_key_iv: "ee8c9e6105840fa41777e787be1ebc61"
server_key_version: "encryptionkey@0"
I20260620 07:09:40.656250 9507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "6f77b1d76e304573a962e43efb94960e"
format_stamp: "Formatted at 2026-06-20 07:09:40 on dist-test-slave-4n6h"
server_key: "0bd0ee295421a708e68f20c9104a6724"
server_key_iv: "ee8c9e6105840fa41777e787be1ebc61"
server_key_version: "encryptionkey@0"
I20260620 07:09:40.660977 9507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 07:09:40.664229 9522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:40.665443 9507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:09:40.665601 9507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "6f77b1d76e304573a962e43efb94960e"
format_stamp: "Formatted at 2026-06-20 07:09:40 on dist-test-slave-4n6h"
server_key: "0bd0ee295421a708e68f20c9104a6724"
server_key_iv: "ee8c9e6105840fa41777e787be1ebc61"
server_key_version: "encryptionkey@0"
I20260620 07:09:40.665789 9507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:40.683312 9507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:40.687136 9507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:40.687485 9507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:40.688342 9507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:40.689612 9507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:40.689682 9507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:40.689843 9507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:40.689908 9507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:40.721177 9507 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:36357
I20260620 07:09:40.721221 9634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:36357 every 8 connection(s)
I20260620 07:09:40.722553 9507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:09:40.728461 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 9507
I20260620 07:09:40.728658 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:09:40.729439 6357 external_mini_cluster.cc:1428] Setting key 21fac4037e0b8d22cca50ae33a604d0e
Jun 20 07:09:40 dist-test-slave-4n6h krb5kdc[9401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939380, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:40.744004 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:40.724960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:39739 (local address 127.6.53.126:46345)
0620 07:09:40.725392 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:40.725397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:40.726397 (+ 1000us) server_negotiation.cc:407] Connection header received
0620 07:09:40.728193 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:40.728197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:40.728260 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:40.728347 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:40.730334 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:40.731402 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:40.733880 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:40.734307 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:40.734402 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:40.738691 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:40.738696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:40.738699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:40.738729 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:40.741138 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:40.741834 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:40.741839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:40.741843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:40.741957 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:40.742387 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:40.742392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:40.742396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:40.742653 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:40.742803 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:40.743423 (+ 620us) server_negotiation.cc:299] Negotiation successful
0620 07:09:40.743646 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":213,"threads_started":1}
I20260620 07:09:40.745028 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:40.725549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:46345 (local address 127.6.53.65:39739)
0620 07:09:40.726209 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:40.726289 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:40.727886 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:40.728622 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:40.728634 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:40.729357 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:40.730124 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:40.730144 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:40.731575 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:40.731580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:40.733745 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:40.733763 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:40.734518 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:40.734527 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:40.734867 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:40.735577 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:40.735615 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:40.738478 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:40.741287 (+ 2809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:40.741295 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:40.741311 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:40.741704 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:40.742058 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:40.742064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:40.742068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:40.742286 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:40.742780 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:40.742788 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:40.743094 (+ 306us) client_negotiation.cc:769] Sending connection context
0620 07:09:40.743342 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 07:09:40.743860 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":262,"threads_started":1}
I20260620 07:09:40.746685 9635 heartbeater.cc:347] Connected to a master server at 127.6.53.126:46345
I20260620 07:09:40.746979 9635 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:40.747740 9635 heartbeater.cc:511] Master 127.6.53.126:46345 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
I20260620 07:09:40.749768 9446 ts_manager.cc:194] Registered new tserver with Master: 6f77b1d76e304573a962e43efb94960e (127.6.53.65:36357)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
I20260620 07:09:40.751483 9446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:39739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:09:40.771304 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:46345
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:41.079826 9641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:41.080081 9641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:41.080272 9641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:41.087394 9641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:41.087541 9641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:41.087739 9641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:09:41.098188 9641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:46345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:41.099534 9641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:41.100880 9641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:41.113849 9647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:41.113924 9646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:41.113961 9649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:41.114527 9641 server_base.cc:1034] running on GCE node
I20260620 07:09:41.115119 9641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:41.116082 9641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:41.117305 9641 hybrid_clock.cc:648] HybridClock initialized: now 1781939381117258 us; error 39 us; skew 500 ppm
I20260620 07:09:41.117611 9641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:41 dist-test-slave-4n6h krb5kdc[9401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939381, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:41.120718 9641 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:09:41.122381 9641 webserver.cc:460] Webserver started at http://127.6.53.66:33021/ using document root <none> and password file <none>
I20260620 07:09:41.123099 9641 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:41.123232 9641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:41.123530 9641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:41.126407 9641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "14ae599ceb39454eb3eb796b1b593c47"
format_stamp: "Formatted at 2026-06-20 07:09:41 on dist-test-slave-4n6h"
server_key: "edfe073bf493cd9df26b2c85d6a61e34"
server_key_iv: "c39fe20467316ab1c83b841a5349c39b"
server_key_version: "encryptionkey@0"
I20260620 07:09:41.127187 9641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "14ae599ceb39454eb3eb796b1b593c47"
format_stamp: "Formatted at 2026-06-20 07:09:41 on dist-test-slave-4n6h"
server_key: "edfe073bf493cd9df26b2c85d6a61e34"
server_key_iv: "c39fe20467316ab1c83b841a5349c39b"
server_key_version: "encryptionkey@0"
I20260620 07:09:41.132316 9641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:09:41.135711 9656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:41.137013 9641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:09:41.137181 9641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "14ae599ceb39454eb3eb796b1b593c47"
format_stamp: "Formatted at 2026-06-20 07:09:41 on dist-test-slave-4n6h"
server_key: "edfe073bf493cd9df26b2c85d6a61e34"
server_key_iv: "c39fe20467316ab1c83b841a5349c39b"
server_key_version: "encryptionkey@0"
I20260620 07:09:41.137373 9641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:41.168574 9641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:41.173118 9641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:41.173424 9641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:41.174463 9641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:41.175813 9641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:41.175907 9641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:41.176029 9641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:41.176101 9641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:41.210644 9641 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:45669
I20260620 07:09:41.210690 9768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:45669 every 8 connection(s)
I20260620 07:09:41.211980 9641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:09:41.217375 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 9641
I20260620 07:09:41.217550 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:09:41.218092 6357 external_mini_cluster.cc:1428] Setting key c7d42d11deb9e7b7d84106affc8c341e
Jun 20 07:09:41 dist-test-slave-4n6h krb5kdc[9401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939381, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:41.233585 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:41.214537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:53675 (local address 127.6.53.126:46345)
0620 07:09:41.214700 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:41.214704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:41.215821 (+ 1117us) server_negotiation.cc:407] Connection header received
0620 07:09:41.217939 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:41.217943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:41.218012 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:41.218103 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:41.219788 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:41.220679 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:41.222284 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:41.222684 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:41.222814 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:41.226474 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:41.226483 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:41.226491 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:41.226536 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:41.230218 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:41.230947 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:41.230955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:41.230961 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:41.231124 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:41.231556 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:41.231564 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:41.231570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:41.231972 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:41.232135 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:41.232923 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 07:09:41.233210 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:09:41.234588 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:41.214990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:46345 (local address 127.6.53.66:53675)
0620 07:09:41.215666 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:41.215702 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:41.217632 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:41.218400 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:41.218412 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:41.219008 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:41.219634 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:41.219650 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:41.220819 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:41.220829 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:41.222119 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:41.222131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:41.222897 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:41.222905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:41.223121 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:41.223812 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:41.223834 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:41.226201 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:41.230377 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:41.230385 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:41.230399 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:41.230816 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:41.231236 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:41.231240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:41.231242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:41.231436 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:41.232126 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:41.232134 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:41.232518 (+ 384us) client_negotiation.cc:769] Sending connection context
0620 07:09:41.232761 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 07:09:41.233247 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":287,"threads_started":1}
I20260620 07:09:41.236100 9769 heartbeater.cc:347] Connected to a master server at 127.6.53.126:46345
I20260620 07:09:41.236502 9769 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
I20260620 07:09:41.237254 9769 heartbeater.cc:511] Master 127.6.53.126:46345 requested a full tablet report, sending...
I20260620 07:09:41.239043 9446 ts_manager.cc:194] Registered new tserver with Master: 14ae599ceb39454eb3eb796b1b593c47 (127.6.53.66:45669)
I20260620 07:09:41.239892 9446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:53675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:09:41.263021 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:46345
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:41.583325 9774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:41.583598 9774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:41.583760 9774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:41.591008 9774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:41.591120 9774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:41.591279 9774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:09:41.601015 9774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:46345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:41.602227 9774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:41.603528 9774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:41.616218 9782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:41.616165 9779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:41.616266 9780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:41.617049 9774 server_base.cc:1034] running on GCE node
I20260620 07:09:41.617626 9774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:41.618597 9774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:41.619817 9774 hybrid_clock.cc:648] HybridClock initialized: now 1781939381619761 us; error 47 us; skew 500 ppm
I20260620 07:09:41.620131 9774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:41 dist-test-slave-4n6h krb5kdc[9401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939381, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:41.623191 9774 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:09:41.624540 9774 webserver.cc:460] Webserver started at http://127.6.53.67:36925/ using document root <none> and password file <none>
I20260620 07:09:41.625142 9774 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:41.625248 9774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:41.625479 9774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:41.628230 9774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "343d52cfde334df4b4fadd400dee3d8c"
format_stamp: "Formatted at 2026-06-20 07:09:41 on dist-test-slave-4n6h"
server_key: "8c0d98f2199a631cc9287066d2b2b5e0"
server_key_iv: "21ba98919ca661d53114ade56102ec77"
server_key_version: "encryptionkey@0"
I20260620 07:09:41.628943 9774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "343d52cfde334df4b4fadd400dee3d8c"
format_stamp: "Formatted at 2026-06-20 07:09:41 on dist-test-slave-4n6h"
server_key: "8c0d98f2199a631cc9287066d2b2b5e0"
server_key_iv: "21ba98919ca661d53114ade56102ec77"
server_key_version: "encryptionkey@0"
I20260620 07:09:41.633702 9774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260620 07:09:41.637122 9789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:41.638554 9774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:09:41.638739 9774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "343d52cfde334df4b4fadd400dee3d8c"
format_stamp: "Formatted at 2026-06-20 07:09:41 on dist-test-slave-4n6h"
server_key: "8c0d98f2199a631cc9287066d2b2b5e0"
server_key_iv: "21ba98919ca661d53114ade56102ec77"
server_key_version: "encryptionkey@0"
I20260620 07:09:41.638942 9774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:41.655412 9774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:41.659291 9774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:41.659556 9774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:41.660442 9774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:41.661731 9774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:41.661808 9774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:41.661914 9774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:41.661965 9774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:41.696203 9774 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:35255
I20260620 07:09:41.696234 9901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:35255 every 8 connection(s)
I20260620 07:09:41.697510 9774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:09:41.701035 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 9774
I20260620 07:09:41.701241 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:09:41.701761 6357 external_mini_cluster.cc:1428] Setting key a627b2d833b04936e3025a4cf8989fca
Jun 20 07:09:41 dist-test-slave-4n6h krb5kdc[9401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939381, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:41.717746 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:41.700049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:42147 (local address 127.6.53.126:46345)
0620 07:09:41.700257 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:41.700262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:41.701643 (+ 1381us) server_negotiation.cc:407] Connection header received
0620 07:09:41.703945 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:41.703951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:41.704055 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:41.704173 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:41.705795 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:41.706644 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:41.708234 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:41.708654 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:41.708794 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:41.712265 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:41.712271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:41.712274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:41.712306 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:41.714862 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:41.715619 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:41.715629 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:41.715632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:41.715790 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:41.716261 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:41.716267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:41.716271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:41.716612 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:41.716746 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:41.717363 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 07:09:41.717517 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:09:41.718781 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:41.700606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:46345 (local address 127.6.53.67:42147)
0620 07:09:41.701487 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:41.701526 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:41.703639 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:41.704412 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:41.704424 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:41.705026 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:41.705613 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:41.705631 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:41.706805 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:41.706811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:41.708065 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:41.708078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:41.708877 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:41.708887 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:41.709156 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:41.709777 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:41.709800 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:41.712043 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:41.714997 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:41.715005 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:41.715021 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:41.715494 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:41.715897 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:41.715901 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:41.715903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:41.716159 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:41.716739 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:41.716747 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:41.717064 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 07:09:41.717289 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 07:09:41.717644 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":365,"threads_started":1}
I20260620 07:09:41.720129 9902 heartbeater.cc:347] Connected to a master server at 127.6.53.126:46345
I20260620 07:09:41.720422 9902 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:41.721186 9902 heartbeater.cc:511] Master 127.6.53.126:46345 requested a full tablet report, sending...
I20260620 07:09:41.722599 9446 ts_manager.cc:194] Registered new tserver with Master: 343d52cfde334df4b4fadd400dee3d8c (127.6.53.67:35255)
I20260620 07:09:41.723364 9446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:42147
I20260620 07:09:41.728933 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 07:09:41 dist-test-slave-4n6h krb5kdc[9401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939381, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260620 07:09:41.754999 9635 heartbeater.cc:503] Master 127.6.53.126:46345 was elected leader, sending a full tablet report...
Jun 20 07:09:41 dist-test-slave-4n6h krb5kdc[9401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939381, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:41.775393 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:41.761255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.6.53.126:46345)
0620 07:09:41.761464 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:41.761473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:41.761899 (+ 426us) server_negotiation.cc:407] Connection header received
0620 07:09:41.762146 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:41.762150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:41.762216 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:41.762376 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:41.763739 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:41.764502 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:41.765926 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:41.766407 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:41.766536 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:41.770240 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:41.770277 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:41.770285 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:41.770331 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:41.772641 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:41.773246 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:41.773252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:41.773255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:41.773404 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:41.773878 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:41.773884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:41.773888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:41.774204 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:41.774497 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:41.774858 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 07:09:41.775080 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260620 07:09:41.775957 9446 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49702
I20260620 07:09:41.777441 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 9507
Jun 20 07:09:41 dist-test-slave-4n6h krb5kdc[9401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939381, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:09:41.793350 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:41.778535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53246 (local address 127.6.53.65:36357)
0620 07:09:41.779215 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:41.779225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:41.779438 (+ 213us) server_negotiation.cc:407] Connection header received
0620 07:09:41.779602 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:41.779611 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:41.779857 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:41.780013 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:41.781156 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:41.782164 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:41.783371 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:41.783971 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:41.784101 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:41.787774 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:41.787784 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:41.787796 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:41.787830 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:41.790759 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:41.791394 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:41.791401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:41.791404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:41.791533 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:41.791874 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:41.791878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:41.791881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:41.792204 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:41.792447 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:41.792744 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 07:09:41.793001 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":324,"threads_started":1}
W20260620 07:09:41.794588 9630 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53246
W20260620 07:09:41.795660 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 9507: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:09:41.795876 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 9507
I20260620 07:09:41.811236 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 9641
Jun 20 07:09:41 dist-test-slave-4n6h krb5kdc[9401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939381, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:09:41.825882 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:41.812077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.6.53.66:45669)
0620 07:09:41.812634 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:41.812640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:41.812666 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:09:41.812723 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:41.812728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:41.812949 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:41.813079 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:41.814175 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:41.815229 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:41.816437 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:41.816962 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:41.817061 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:41.820952 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:41.820963 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:41.820976 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:41.821009 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:41.823669 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:41.824245 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:41.824251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:41.824253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:41.824382 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:41.824667 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:41.824670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:41.824672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:41.824982 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:41.825200 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:41.825404 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 07:09:41.825581 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":319,"threads_started":1}
W20260620 07:09:41.827241 9764 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53948
W20260620 07:09:41.828315 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 9641: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:09:41.828437 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 9641
I20260620 07:09:41.844954 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 9774
Jun 20 07:09:41 dist-test-slave-4n6h krb5kdc[9401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939381, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:09:41.860966 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:41.845927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38038 (local address 127.6.53.67:35255)
0620 07:09:41.846586 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:41.846593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:41.846620 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:09:41.846682 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:41.846688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:41.846933 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:41.847076 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:41.848350 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:41.849410 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:41.850835 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:41.851480 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:41.851608 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:41.855925 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:41.855936 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:41.855949 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:41.855981 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:41.858618 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:41.859226 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:41.859232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:41.859235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:41.859369 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:41.859651 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:41.859654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:41.859656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:41.859954 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:41.860186 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:41.860393 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 07:09:41.860573 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":369,"threads_started":1}
W20260620 07:09:41.862295 9897 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38038
W20260620 07:09:41.863483 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 9774: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:09:41.863641 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 9774
I20260620 07:09:41.878513 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 9414
I20260620 07:09:41.879602 9476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:41.979494 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 9414
2026-06-20T07:09:41Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3487 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:09:42 dist-test-slave-4n6h krb5kdc[9919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:09:42 dist-test-slave-4n6h krb5kdc[9919](info): set up 2 sockets
Jun 20 07:09:42 dist-test-slave-4n6h krb5kdc[9919](info): commencing operation
krb5kdc: starting...
W20260620 07:09:43.099287 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:09:43 dist-test-slave-4n6h krb5kdc[9919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:09:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:09:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:09:43.286243 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:38057
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:38057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:43.608914 9932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:43.609359 9932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:43.609546 9932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:43.620873 9932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:43.620987 9932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:43.621034 9932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:43.621091 9932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:43.621134 9932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:43.630961 9932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:38057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:38057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:43.632211 9932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:43.633455 9932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:43.644747 9937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:43.644683 9938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:43.644812 9940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:43.645582 9932 server_base.cc:1034] running on GCE node
I20260620 07:09:43.646414 9932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:43.647802 9932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:43.648988 9932 hybrid_clock.cc:648] HybridClock initialized: now 1781939383648932 us; error 48 us; skew 500 ppm
I20260620 07:09:43.649312 9932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:43 dist-test-slave-4n6h krb5kdc[9919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939383, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:43.652381 9932 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:09:43.653643 9932 webserver.cc:460] Webserver started at http://127.6.53.126:44861/ using document root <none> and password file <none>
I20260620 07:09:43.654242 9932 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:43.654395 9932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:43.654661 9932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:43.657371 9932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "3836728077904b11a6048a789582b9c1"
format_stamp: "Formatted at 2026-06-20 07:09:43 on dist-test-slave-4n6h"
server_key: "4b24fc1bc328f8fb4b46c2f4ad120322"
server_key_iv: "aa3fe962b1bc36e4af434d567365c4c8"
server_key_version: "encryptionkey@0"
I20260620 07:09:43.658097 9932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "3836728077904b11a6048a789582b9c1"
format_stamp: "Formatted at 2026-06-20 07:09:43 on dist-test-slave-4n6h"
server_key: "4b24fc1bc328f8fb4b46c2f4ad120322"
server_key_iv: "aa3fe962b1bc36e4af434d567365c4c8"
server_key_version: "encryptionkey@0"
I20260620 07:09:43.662966 9932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 07:09:43.665994 9947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:43.667254 9932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:09:43.667384 9932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "3836728077904b11a6048a789582b9c1"
format_stamp: "Formatted at 2026-06-20 07:09:43 on dist-test-slave-4n6h"
server_key: "4b24fc1bc328f8fb4b46c2f4ad120322"
server_key_iv: "aa3fe962b1bc36e4af434d567365c4c8"
server_key_version: "encryptionkey@0"
I20260620 07:09:43.667531 9932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:43.698678 9932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:43.702392 9932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:43.702682 9932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:43.720340 9932 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:38057
I20260620 07:09:43.720373 9998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:38057 every 8 connection(s)
I20260620 07:09:43.721621 9932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:43.723186 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 9932
I20260620 07:09:43.723400 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:09:43.724035 6357 external_mini_cluster.cc:1428] Setting key 610ed631e902d2d1616ce8de87382908
I20260620 07:09:43.725466 9999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 07:09:43 dist-test-slave-4n6h krb5kdc[9919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:43.738960 9999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1: Bootstrap starting.
I20260620 07:09:43.742152 9999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:43.743389 9999 log.cc:824] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:43.745818 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:43.725884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.6.53.126:38057)
0620 07:09:43.726648 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:43.726660 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:43.726698 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:09:43.728186 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:43.728204 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:43.728565 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:43.728872 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:43.730627 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:43.731898 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:43.733586 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:43.734192 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:43.734382 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:43.739090 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:43.739100 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:43.739118 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:43.739162 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:43.741965 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:43.742765 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:43.742771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:43.742777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:43.742960 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:43.743360 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:43.743363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:43.743365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:43.743967 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:43.744306 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:43.744620 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 07:09:43.744943 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":379,"threads_started":1}
I20260620 07:09:43.746317 9999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1: No bootstrap required, opened a new log
I20260620 07:09:43.753867 9999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3836728077904b11a6048a789582b9c1" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38057 } }
I20260620 07:09:43.754316 9999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:43.754457 9999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3836728077904b11a6048a789582b9c1, State: Initialized, Role: FOLLOWER
I20260620 07:09:43.755115 9999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3836728077904b11a6048a789582b9c1" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38057 } }
I20260620 07:09:43.755347 9999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:09:43.755496 9999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:09:43.755659 9999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:43.757769 9999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3836728077904b11a6048a789582b9c1" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38057 } }
I20260620 07:09:43.758291 9999 leader_election.cc:302] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3836728077904b11a6048a789582b9c1; no voters:
I20260620 07:09:43.758942 9999 leader_election.cc:288] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:09:43.759071 10004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:43.759771 10004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [term 1 LEADER]: Becoming Leader. State: Replica: 3836728077904b11a6048a789582b9c1, State: Running, Role: LEADER
I20260620 07:09:43.760262 10004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3836728077904b11a6048a789582b9c1" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38057 } }
I20260620 07:09:43.760929 9999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:43.763072 10006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3836728077904b11a6048a789582b9c1. Latest consensus state: current_term: 1 leader_uuid: "3836728077904b11a6048a789582b9c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3836728077904b11a6048a789582b9c1" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38057 } } }
I20260620 07:09:43.763262 10005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3836728077904b11a6048a789582b9c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3836728077904b11a6048a789582b9c1" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38057 } } }
I20260620 07:09:43.763587 10006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:43.763712 10005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:43.765106 10013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:09:43.770052 10013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:09:43.778093 10013 catalog_manager.cc:1347] Generated new cluster ID: 2178c6c7420c4ef4b6b903e6197e3063
I20260620 07:09:43.778216 10013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:43.793200 10013 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:43.794577 10013 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:43.801962 10013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3836728077904b11a6048a789582b9c1: Generated new TSK 0
I20260620 07:09:43.802893 10013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:09:43.861027 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38057
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:44.180457 10025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:44.180739 10025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:44.180918 10025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:44.188095 10025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:44.188213 10025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:44.188382 10025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:09:44.198598 10025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:44.199811 10025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:44.201109 10025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:44.214895 10031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:44.214993 10030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:44.214954 10033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:44.215824 10025 server_base.cc:1034] running on GCE node
I20260620 07:09:44.216406 10025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:44.217388 10025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:44.218616 10025 hybrid_clock.cc:648] HybridClock initialized: now 1781939384218567 us; error 54 us; skew 500 ppm
I20260620 07:09:44.218923 10025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:44 dist-test-slave-4n6h krb5kdc[9919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939384, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:44.221817 10025 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:09:44.223147 10025 webserver.cc:460] Webserver started at http://127.6.53.65:35025/ using document root <none> and password file <none>
I20260620 07:09:44.223749 10025 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:44.223845 10025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:44.224092 10025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:44.226847 10025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "87489d2d04c0433780be0d0f5352c6e8"
format_stamp: "Formatted at 2026-06-20 07:09:44 on dist-test-slave-4n6h"
server_key: "c65f425e7bd9e1c96e92e28d2eee13be"
server_key_iv: "11428689b672806feb8861841cc2fe2e"
server_key_version: "encryptionkey@0"
I20260620 07:09:44.227536 10025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "87489d2d04c0433780be0d0f5352c6e8"
format_stamp: "Formatted at 2026-06-20 07:09:44 on dist-test-slave-4n6h"
server_key: "c65f425e7bd9e1c96e92e28d2eee13be"
server_key_iv: "11428689b672806feb8861841cc2fe2e"
server_key_version: "encryptionkey@0"
I20260620 07:09:44.232271 10025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 07:09:44.235396 10040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:44.236599 10025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:44.236752 10025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "87489d2d04c0433780be0d0f5352c6e8"
format_stamp: "Formatted at 2026-06-20 07:09:44 on dist-test-slave-4n6h"
server_key: "c65f425e7bd9e1c96e92e28d2eee13be"
server_key_iv: "11428689b672806feb8861841cc2fe2e"
server_key_version: "encryptionkey@0"
I20260620 07:09:44.236941 10025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:44.253969 10025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:44.257767 10025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:44.258044 10025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:44.259006 10025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:44.260315 10025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:44.260413 10025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:44.260567 10025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:44.260635 10025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:44.292676 10025 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:35957
I20260620 07:09:44.292717 10152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:35957 every 8 connection(s)
I20260620 07:09:44.293912 10025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:09:44.298705 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 10025
I20260620 07:09:44.298923 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:09:44.299525 6357 external_mini_cluster.cc:1428] Setting key ec75687451f3cbe344b8c8a704c43994
Jun 20 07:09:44 dist-test-slave-4n6h krb5kdc[9919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939384, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:44.313289 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:44.296497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:47775 (local address 127.6.53.126:38057)
0620 07:09:44.296953 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:44.296959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:44.297934 (+ 975us) server_negotiation.cc:407] Connection header received
0620 07:09:44.299800 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:44.299806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:44.299894 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:44.300018 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:44.301648 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:44.302425 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:44.303896 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:44.304498 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:44.304630 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:44.307883 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:44.307889 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:44.307894 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:44.307939 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:44.310463 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:44.311128 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:44.311137 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:44.311142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:44.311265 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:44.311669 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:44.311676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:44.311680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:44.311932 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:44.312077 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:44.312721 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 07:09:44.312935 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":210,"threads_started":1}
I20260620 07:09:44.314215 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:44.297116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:38057 (local address 127.6.53.65:47775)
0620 07:09:44.297785 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:44.297826 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:44.299512 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:44.300239 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:44.300250 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:44.300790 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:44.301486 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:44.301504 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:44.302563 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:44.302572 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:44.303751 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:44.303763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:44.304710 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:44.304718 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:44.304935 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:44.305594 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:44.305618 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:44.307689 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:44.310591 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:44.310599 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:44.310614 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:44.311014 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:44.311367 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:44.311373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:44.311377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:44.311566 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:44.312053 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:44.312061 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:44.312375 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 07:09:44.312639 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 07:09:44.313143 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":268,"threads_started":1}
I20260620 07:09:44.315847 10153 heartbeater.cc:347] Connected to a master server at 127.6.53.126:38057
I20260620 07:09:44.316124 10153 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
I20260620 07:09:44.316803 10153 heartbeater.cc:511] Master 127.6.53.126:38057 requested a full tablet report, sending...
I20260620 07:09:44.318926 9964 ts_manager.cc:194] Registered new tserver with Master: 87489d2d04c0433780be0d0f5352c6e8 (127.6.53.65:35957)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
I20260620 07:09:44.320461 9964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:47775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:09:44.341588 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38057
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:44.657179 10159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:44.657466 10159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:44.657573 10159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:44.665344 10159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:44.665480 10159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:44.665625 10159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:09:44.675525 10159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:44.676837 10159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:44.678092 10159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:44.690951 10167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:44.691018 10164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:44.690922 10165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:44.692040 10159 server_base.cc:1034] running on GCE node
I20260620 07:09:44.692612 10159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:44.693576 10159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:44.694784 10159 hybrid_clock.cc:648] HybridClock initialized: now 1781939384694737 us; error 45 us; skew 500 ppm
I20260620 07:09:44.695084 10159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:44 dist-test-slave-4n6h krb5kdc[9919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939384, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:44.698158 10159 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:09:44.699539 10159 webserver.cc:460] Webserver started at http://127.6.53.66:46727/ using document root <none> and password file <none>
I20260620 07:09:44.700155 10159 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:44.700253 10159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:44.700477 10159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:44.703277 10159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "61be42d1eb694f0aa89316f1435b54dd"
format_stamp: "Formatted at 2026-06-20 07:09:44 on dist-test-slave-4n6h"
server_key: "026aa0615a38a7cc7092a68134ce0d96"
server_key_iv: "e881524c4eef54a7b1e0b1dfd20dd302"
server_key_version: "encryptionkey@0"
I20260620 07:09:44.703975 10159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "61be42d1eb694f0aa89316f1435b54dd"
format_stamp: "Formatted at 2026-06-20 07:09:44 on dist-test-slave-4n6h"
server_key: "026aa0615a38a7cc7092a68134ce0d96"
server_key_iv: "e881524c4eef54a7b1e0b1dfd20dd302"
server_key_version: "encryptionkey@0"
I20260620 07:09:44.708850 10159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:09:44.712116 10174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:44.713387 10159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:09:44.713552 10159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "61be42d1eb694f0aa89316f1435b54dd"
format_stamp: "Formatted at 2026-06-20 07:09:44 on dist-test-slave-4n6h"
server_key: "026aa0615a38a7cc7092a68134ce0d96"
server_key_iv: "e881524c4eef54a7b1e0b1dfd20dd302"
server_key_version: "encryptionkey@0"
I20260620 07:09:44.713752 10159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:44.733335 10159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:44.737140 10159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:44.737430 10159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:44.738493 10159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:44.740077 10159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:44.740170 10159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:44.740315 10159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:44.740381 10159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:44.772929 10159 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:35409
I20260620 07:09:44.773026 10286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:35409 every 8 connection(s)
I20260620 07:09:44.774184 10159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:09:44.778329 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 10159
I20260620 07:09:44.778534 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:09:44.778951 6357 external_mini_cluster.cc:1428] Setting key 28408a4b70128de65ab88cab1ee427bc
Jun 20 07:09:44 dist-test-slave-4n6h krb5kdc[9919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939384, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:44.794004 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:44.776869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:42281 (local address 127.6.53.126:38057)
0620 07:09:44.777062 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:44.777067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:44.778181 (+ 1114us) server_negotiation.cc:407] Connection header received
0620 07:09:44.780439 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:44.780445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:44.780537 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:44.780650 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:44.782301 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:44.783010 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:44.784499 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:44.784908 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:44.785036 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:44.788578 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:44.788583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:44.788586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:44.788618 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:44.791171 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:44.791844 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:44.791849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:44.791853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:44.791979 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:44.792389 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:44.792394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:44.792397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:44.792663 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:44.792809 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:44.793437 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 07:09:44.793658 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:09:44.794971 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:44.777299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:38057 (local address 127.6.53.66:42281)
0620 07:09:44.778011 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:44.778049 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:44.780177 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:44.780905 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:44.780913 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:44.781456 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:44.782080 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:44.782096 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:44.783163 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:44.783167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:44.784362 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:44.784374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:44.785125 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:44.785131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:44.785350 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:44.785999 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:44.786025 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:44.788365 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:44.791303 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:44.791311 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:44.791326 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:44.791729 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:44.792093 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:44.792098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:44.792102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:44.792276 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:44.792784 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:44.792791 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:44.793108 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 07:09:44.793363 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 07:09:44.793889 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":282,"threads_started":1}
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
I20260620 07:09:44.796519 10287 heartbeater.cc:347] Connected to a master server at 127.6.53.126:38057
I20260620 07:09:44.796839 10287 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:44.797710 10287 heartbeater.cc:511] Master 127.6.53.126:38057 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
I20260620 07:09:44.799316 9964 ts_manager.cc:194] Registered new tserver with Master: 61be42d1eb694f0aa89316f1435b54dd (127.6.53.66:35409)
I20260620 07:09:44.800601 9964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:42281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:09:44.819612 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38057
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:45.128458 10292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:45.128705 10292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:45.128793 10292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:45.135680 10292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:45.135788 10292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:45.135901 10292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:09:45.145323 10292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:45.146449 10292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:45.147641 10292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:45.160063 10298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:45.160025 10300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:45.160241 10297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:45.160893 10292 server_base.cc:1034] running on GCE node
I20260620 07:09:45.161468 10292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:45.162449 10292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:45.163705 10292 hybrid_clock.cc:648] HybridClock initialized: now 1781939385163644 us; error 45 us; skew 500 ppm
I20260620 07:09:45.164014 10292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:45 dist-test-slave-4n6h krb5kdc[9919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939385, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:45.166955 10292 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:09:45.168349 10292 webserver.cc:460] Webserver started at http://127.6.53.67:40639/ using document root <none> and password file <none>
I20260620 07:09:45.168973 10292 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:45.169106 10292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:45.169384 10292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:45.172233 10292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "8962cd402e0741658f6bd782f9fc2577"
format_stamp: "Formatted at 2026-06-20 07:09:45 on dist-test-slave-4n6h"
server_key: "f3819babe8953596749f9f8c4cbe8205"
server_key_iv: "577b6e52a02922f94525545c26088554"
server_key_version: "encryptionkey@0"
I20260620 07:09:45.172973 10292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "8962cd402e0741658f6bd782f9fc2577"
format_stamp: "Formatted at 2026-06-20 07:09:45 on dist-test-slave-4n6h"
server_key: "f3819babe8953596749f9f8c4cbe8205"
server_key_iv: "577b6e52a02922f94525545c26088554"
server_key_version: "encryptionkey@0"
I20260620 07:09:45.177664 10292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 07:09:45.180794 10307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:45.181973 10292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:09:45.182128 10292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "8962cd402e0741658f6bd782f9fc2577"
format_stamp: "Formatted at 2026-06-20 07:09:45 on dist-test-slave-4n6h"
server_key: "f3819babe8953596749f9f8c4cbe8205"
server_key_iv: "577b6e52a02922f94525545c26088554"
server_key_version: "encryptionkey@0"
I20260620 07:09:45.182319 10292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:45.215530 10292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:45.219691 10292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:45.219992 10292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:45.220897 10292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:45.222155 10292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:45.222250 10292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:45.222417 10292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:45.222484 10292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:45.254388 10292 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:46335
I20260620 07:09:45.254407 10419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:46335 every 8 connection(s)
I20260620 07:09:45.255625 10292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:09:45.255940 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 10292
I20260620 07:09:45.256134 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:09:45.256649 6357 external_mini_cluster.cc:1428] Setting key d9abb181c2bf1fbc5eb5b5a66694a82f
Jun 20 07:09:45 dist-test-slave-4n6h krb5kdc[9919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939385, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:45.275200 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:45.258243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:60341 (local address 127.6.53.126:38057)
0620 07:09:45.258698 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:45.258703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:45.259974 (+ 1271us) server_negotiation.cc:407] Connection header received
0620 07:09:45.261712 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:45.261716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:45.261815 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:45.261924 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:45.263418 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:45.264213 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:45.265759 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:45.266165 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:45.266310 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:45.269551 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:45.269556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:45.269559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:45.269589 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:45.272738 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:45.273377 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:45.273380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:45.273383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:45.273497 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:45.273829 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:45.273832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:45.273834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:45.274075 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:45.274175 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:45.274809 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 07:09:45.274962 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282}
I20260620 07:09:45.276077 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:45.259017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:38057 (local address 127.6.53.67:60341)
0620 07:09:45.259823 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:45.259882 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:45.261434 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:45.262156 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:45.262164 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:45.262731 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:45.263261 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:45.263278 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:45.264352 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:45.264357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:45.265625 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:45.265636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:45.266403 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:45.266411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:45.266651 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:45.267228 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:45.267255 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:45.269329 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:45.272873 (+ 3544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:45.272879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:45.272890 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:45.273276 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:45.273579 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:45.273582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:45.273584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:45.273739 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:45.274169 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:45.274174 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:45.274512 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 07:09:45.274722 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 07:09:45.275081 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":316,"threads_started":1}
I20260620 07:09:45.277316 10420 heartbeater.cc:347] Connected to a master server at 127.6.53.126:38057
I20260620 07:09:45.277603 10420 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:45.278326 10420 heartbeater.cc:511] Master 127.6.53.126:38057 requested a full tablet report, sending...
I20260620 07:09:45.279862 9964 ts_manager.cc:194] Registered new tserver with Master: 8962cd402e0741658f6bd782f9fc2577 (127.6.53.67:46335)
I20260620 07:09:45.280660 9964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:60341
I20260620 07:09:45.284293 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 07:09:45 dist-test-slave-4n6h krb5kdc[9919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260620 07:09:45.323936 10153 heartbeater.cc:503] Master 127.6.53.126:38057 was elected leader, sending a full tablet report...
Jun 20 07:09:45 dist-test-slave-4n6h krb5kdc[9919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:09:45.330786 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:45.316341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35798 (local address 127.6.53.65:35957)
0620 07:09:45.317015 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:45.317024 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:45.317059 (+ 35us) server_negotiation.cc:407] Connection header received
0620 07:09:45.317124 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:45.317130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:45.317337 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:45.317473 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:45.318875 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:45.319944 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:45.321129 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:45.321677 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:45.321775 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:45.325745 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:45.325759 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:45.325773 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:45.325803 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:45.328486 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:45.329037 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:45.329043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:45.329045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:45.329156 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:45.329475 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:45.329478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:45.329480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:45.329783 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:45.330000 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:45.330172 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 07:09:45.330406 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":334,"threads_started":1}
Jun 20 07:09:45 dist-test-slave-4n6h krb5kdc[9919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:45.347082 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:45.334714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44406 (local address 127.6.53.126:38057)
0620 07:09:45.334934 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:45.334942 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:45.335035 (+ 93us) server_negotiation.cc:407] Connection header received
0620 07:09:45.335154 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:45.335157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:45.335214 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:45.335290 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:45.336392 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:45.337054 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:45.338162 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:45.338596 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:45.338684 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:45.342337 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:45.342343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:45.342347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:45.342390 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:45.344791 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:45.345329 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:45.345332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:45.345334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:45.345461 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:45.345823 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:45.345827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:45.345830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:45.346169 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:45.346490 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:45.346681 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 07:09:45.346849 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 07:09:45.347648 9964 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44406
I20260620 07:09:45.348662 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 10025
I20260620 07:09:45.360658 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:45.349576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.6.53.65:35957)
0620 07:09:45.349773 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:45.349778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:45.349845 (+ 67us) server_negotiation.cc:407] Connection header received
0620 07:09:45.350024 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:45.350030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:45.350097 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:45.350184 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:45.351324 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:45.352120 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:45.353266 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:45.353686 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:45.353790 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:45.355918 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:45.355922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:45.355925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:45.355956 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:45.358587 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:45.359113 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:45.359117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:45.359119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:45.359231 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:45.359515 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:45.359518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:45.359520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:45.359742 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:45.359914 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:45.360141 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 07:09:45.360289 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:09:45.361294 10148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:45.441133 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 10025
I20260620 07:09:45.458508 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 10159
Jun 20 07:09:45 dist-test-slave-4n6h krb5kdc[9919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:09:45.473564 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:45.459356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.6.53.66:35409)
0620 07:09:45.459883 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:45.459890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:45.459915 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:09:45.459979 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:45.459985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:45.460190 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:45.460318 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:45.461476 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:45.462625 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:45.463797 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:45.464356 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:45.464484 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:45.468321 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:45.468331 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:45.468345 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:45.468380 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:45.471258 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:45.471868 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:45.471875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:45.471877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:45.471994 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:45.472311 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:45.472314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:45.472316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:45.472671 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:45.472906 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:45.473110 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 07:09:45.473288 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":265,"threads_started":1}
I20260620 07:09:45.474833 10282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:45.555231 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 10159
I20260620 07:09:45.571717 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 10292
Jun 20 07:09:45 dist-test-slave-4n6h krb5kdc[9919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:09:45.587635 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:45.572554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57022 (local address 127.6.53.67:46335)
0620 07:09:45.573123 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:45.573130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:45.573157 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:09:45.573215 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:45.573220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:45.573422 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:45.573596 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:45.574938 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:45.575954 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:45.577255 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:45.577791 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:45.577937 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:45.582049 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:45.582060 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:45.582074 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:45.582111 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:45.584716 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:45.585441 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:45.585448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:45.585451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:45.585572 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:45.585965 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:45.585974 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:45.585977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:45.586417 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:45.586736 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:45.587043 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 07:09:45.587301 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":277,"threads_started":1}
I20260620 07:09:45.588876 10415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:45.670897 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 10292
I20260620 07:09:45.686728 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 9932
I20260620 07:09:45.687887 9994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:45.788363 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 9932
2026-06-20T07:09:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3808 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:09:45 dist-test-slave-4n6h krb5kdc[10439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:09:45 dist-test-slave-4n6h krb5kdc[10439](info): set up 2 sockets
Jun 20 07:09:45 dist-test-slave-4n6h krb5kdc[10439](info): commencing operation
krb5kdc: starting...
W20260620 07:09:46.910828 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:09:47 dist-test-slave-4n6h krb5kdc[10439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:09:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:09:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:09:47.101783 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:36445
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:41269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:36445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:47.428061 10452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:47.428364 10452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:47.428465 10452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:47.435874 10452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:47.435988 10452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:47.436054 10452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:47.436127 10452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:47.436183 10452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:47.446295 10452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:41269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:36445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:36445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:47.447552 10452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:47.448868 10452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:47.460917 10458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:47.461031 10457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:47.460999 10460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:47.462010 10452 server_base.cc:1034] running on GCE node
I20260620 07:09:47.462890 10452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:47.464351 10452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:47.465603 10452 hybrid_clock.cc:648] HybridClock initialized: now 1781939387465564 us; error 60 us; skew 500 ppm
I20260620 07:09:47.465969 10452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:47 dist-test-slave-4n6h krb5kdc[10439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939387, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:47.469022 10452 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:09:47.470140 10452 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:47.470269 10452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:47.470610 10452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:47.473522 10452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "1200917b6a6c4463b4d589f04c14a996"
format_stamp: "Formatted at 2026-06-20 07:09:47 on dist-test-slave-4n6h"
server_key: "32c455b15e6523f4924c559be6f76e5b"
server_key_iv: "952ed1b7bf2955dbb70571cd77f6700d"
server_key_version: "encryptionkey@0"
I20260620 07:09:47.474375 10452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "1200917b6a6c4463b4d589f04c14a996"
format_stamp: "Formatted at 2026-06-20 07:09:47 on dist-test-slave-4n6h"
server_key: "32c455b15e6523f4924c559be6f76e5b"
server_key_iv: "952ed1b7bf2955dbb70571cd77f6700d"
server_key_version: "encryptionkey@0"
I20260620 07:09:47.479238 10452 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:09:47.482529 10466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:47.483785 10452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:09:47.483995 10452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "1200917b6a6c4463b4d589f04c14a996"
format_stamp: "Formatted at 2026-06-20 07:09:47 on dist-test-slave-4n6h"
server_key: "32c455b15e6523f4924c559be6f76e5b"
server_key_iv: "952ed1b7bf2955dbb70571cd77f6700d"
server_key_version: "encryptionkey@0"
I20260620 07:09:47.484210 10452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:47.500481 10452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:47.504575 10452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:47.504838 10452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:47.522969 10452 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:36445
I20260620 07:09:47.523015 10517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:36445 every 8 connection(s)
I20260620 07:09:47.523826 10452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:47.527324 10518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:47.529794 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 10452
I20260620 07:09:47.530030 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:09:47.530798 6357 external_mini_cluster.cc:1428] Setting key 18ee7f9b744f09deb8667fb1ccdd4471
I20260620 07:09:47.539575 10518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996: Bootstrap starting.
I20260620 07:09:47.543232 10518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:47.544514 10518 log.cc:824] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:09:47 dist-test-slave-4n6h krb5kdc[10439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:47.547503 10518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996: No bootstrap required, opened a new log
I20260620 07:09:47.553637 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:47.532849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.6.53.126:36445)
0620 07:09:47.533637 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:47.533648 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:47.533686 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:09:47.535338 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:47.535359 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:47.535726 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:47.536043 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:47.537829 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:47.539169 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:47.540866 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:47.541523 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:47.541642 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:47.546584 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:47.546591 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:47.546605 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:47.546640 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:47.549702 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:47.550509 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:47.550515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:47.550520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:47.550671 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:47.551046 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:47.551049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:47.551051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:47.551569 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:47.551780 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:47.552229 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 07:09:47.552585 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":381,"threads_started":1}
I20260620 07:09:47.556505 10518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1200917b6a6c4463b4d589f04c14a996" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36445 } }
I20260620 07:09:47.556886 10518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:47.556998 10518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1200917b6a6c4463b4d589f04c14a996, State: Initialized, Role: FOLLOWER
I20260620 07:09:47.557570 10518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1200917b6a6c4463b4d589f04c14a996" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36445 } }
I20260620 07:09:47.557793 10518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:09:47.557934 10518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:09:47.558115 10518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:47.560222 10518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1200917b6a6c4463b4d589f04c14a996" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36445 } }
I20260620 07:09:47.560695 10518 leader_election.cc:302] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1200917b6a6c4463b4d589f04c14a996; no voters:
I20260620 07:09:47.561275 10518 leader_election.cc:288] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:09:47.561434 10523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:47.562479 10523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [term 1 LEADER]: Becoming Leader. State: Replica: 1200917b6a6c4463b4d589f04c14a996, State: Running, Role: LEADER
I20260620 07:09:47.562940 10523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1200917b6a6c4463b4d589f04c14a996" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36445 } }
I20260620 07:09:47.563408 10518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:47.565899 10525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1200917b6a6c4463b4d589f04c14a996. Latest consensus state: current_term: 1 leader_uuid: "1200917b6a6c4463b4d589f04c14a996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1200917b6a6c4463b4d589f04c14a996" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36445 } } }
I20260620 07:09:47.565876 10524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1200917b6a6c4463b4d589f04c14a996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1200917b6a6c4463b4d589f04c14a996" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36445 } } }
I20260620 07:09:47.567438 10525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:47.567447 10524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:47.568912 10533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:09:47.573850 10533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:09:47.582383 10533 catalog_manager.cc:1347] Generated new cluster ID: d20395b0529349c1b182dd5bf21378d0
I20260620 07:09:47.582496 10533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:47.600376 10533 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:47.601641 10533 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:47.610726 10533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996: Generated new TSK 0
I20260620 07:09:47.611729 10533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:09:47.661085 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:36445
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:41269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:47.982169 10544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:47.982443 10544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:47.982546 10544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:47.989789 10544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:47.989897 10544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:47.990012 10544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:09:47.999763 10544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:41269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:36445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:48.001057 10544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:48.002528 10544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:48.015900 10549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:48.015985 10552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:48.015942 10550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:48.016736 10544 server_base.cc:1034] running on GCE node
I20260620 07:09:48.017290 10544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:48.018290 10544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:48.019503 10544 hybrid_clock.cc:648] HybridClock initialized: now 1781939388019443 us; error 57 us; skew 500 ppm
I20260620 07:09:48.019822 10544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:48 dist-test-slave-4n6h krb5kdc[10439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939388, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:48.023021 10544 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:09:48.024166 10544 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:48.024253 10544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:48.024606 10544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:48.027526 10544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "25adde3ed19c432d8afdb4d53a2c832d"
format_stamp: "Formatted at 2026-06-20 07:09:48 on dist-test-slave-4n6h"
server_key: "7dc3108307f1f7680845f63454d81612"
server_key_iv: "3f4f47bee466a22633dfaf004f5dea48"
server_key_version: "encryptionkey@0"
I20260620 07:09:48.028319 10544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "25adde3ed19c432d8afdb4d53a2c832d"
format_stamp: "Formatted at 2026-06-20 07:09:48 on dist-test-slave-4n6h"
server_key: "7dc3108307f1f7680845f63454d81612"
server_key_iv: "3f4f47bee466a22633dfaf004f5dea48"
server_key_version: "encryptionkey@0"
I20260620 07:09:48.033165 10544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 07:09:48.036609 10558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:48.037943 10544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:09:48.038118 10544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "25adde3ed19c432d8afdb4d53a2c832d"
format_stamp: "Formatted at 2026-06-20 07:09:48 on dist-test-slave-4n6h"
server_key: "7dc3108307f1f7680845f63454d81612"
server_key_iv: "3f4f47bee466a22633dfaf004f5dea48"
server_key_version: "encryptionkey@0"
I20260620 07:09:48.038353 10544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:48.061100 10544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:48.065099 10544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:48.065346 10544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:48.066327 10544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:48.067605 10544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:48.067726 10544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:48.067850 10544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:48.067936 10544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:48.100935 10544 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:33751
I20260620 07:09:48.101027 10670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:33751 every 8 connection(s)
I20260620 07:09:48.101814 10544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:09:48.109545 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 10544
I20260620 07:09:48.109788 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:09:48.110500 6357 external_mini_cluster.cc:1428] Setting key 57e93aa92ddbdd42226fdc1e7ef23c38
Jun 20 07:09:48 dist-test-slave-4n6h krb5kdc[10439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939388, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:48.122153 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:48.104362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:43407 (local address 127.6.53.126:36445)
0620 07:09:48.104759 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:48.104763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:48.105778 (+ 1015us) server_negotiation.cc:407] Connection header received
0620 07:09:48.107619 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:48.107625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:48.107710 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:48.107832 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:48.109302 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:48.110214 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:48.111947 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:48.112355 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:48.112490 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:48.116502 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:48.116508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:48.116514 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:48.116553 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:48.119193 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:48.119915 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:48.119922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:48.119925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:48.120055 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:48.120467 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:48.120474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:48.120477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:48.120755 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:48.120909 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:48.121561 (+ 652us) server_negotiation.cc:299] Negotiation successful
0620 07:09:48.121778 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":196,"threads_started":1}
I20260620 07:09:48.123104 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:48.104928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:36445 (local address 127.6.53.65:43407)
0620 07:09:48.105616 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:48.105659 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:48.107354 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:48.108063 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:48.108071 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:48.108571 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:48.109127 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:48.109145 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:48.110426 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:48.110434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:48.111787 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:48.111800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:48.112585 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:48.112592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:48.112864 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:48.113639 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:48.113667 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:48.116286 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:48.119337 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:48.119346 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:48.119363 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:48.119790 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:48.120173 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:48.120178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:48.120181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:48.120355 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:48.120882 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:48.120890 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:48.121226 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 07:09:48.121486 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 07:09:48.122007 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":280,"threads_started":1}
I20260620 07:09:48.124547 10671 heartbeater.cc:347] Connected to a master server at 127.6.53.126:36445
I20260620 07:09:48.124879 10671 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:48.125682 10671 heartbeater.cc:511] Master 127.6.53.126:36445 requested a full tablet report, sending...
I20260620 07:09:48.127943 10483 ts_manager.cc:194] Registered new tserver with Master: 25adde3ed19c432d8afdb4d53a2c832d (127.6.53.65:33751)
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
I20260620 07:09:48.129385 10483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:43407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:09:48.154454 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:36445
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:41269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:48.478449 10677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:48.478745 10677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:48.478865 10677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:48.486950 10677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:48.487062 10677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:48.487188 10677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:09:48.498185 10677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:41269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:36445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:48.499475 10677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:48.500763 10677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:48.514592 10682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:48.514654 10685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:48.514632 10683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:48.515383 10677 server_base.cc:1034] running on GCE node
I20260620 07:09:48.515931 10677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:48.516819 10677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:48.518005 10677 hybrid_clock.cc:648] HybridClock initialized: now 1781939388517947 us; error 51 us; skew 500 ppm
I20260620 07:09:48.518368 10677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:48 dist-test-slave-4n6h krb5kdc[10439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939388, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:48.521373 10677 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:09:48.522606 10677 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:48.522715 10677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:48.523059 10677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:48.525749 10677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "522e935aa7f543c2b12e5840b3afeb8b"
format_stamp: "Formatted at 2026-06-20 07:09:48 on dist-test-slave-4n6h"
server_key: "eb0220770026b09453c5e2a2c9763271"
server_key_iv: "00388201126fa9cf1378304358d17275"
server_key_version: "encryptionkey@0"
I20260620 07:09:48.526638 10677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "522e935aa7f543c2b12e5840b3afeb8b"
format_stamp: "Formatted at 2026-06-20 07:09:48 on dist-test-slave-4n6h"
server_key: "eb0220770026b09453c5e2a2c9763271"
server_key_iv: "00388201126fa9cf1378304358d17275"
server_key_version: "encryptionkey@0"
I20260620 07:09:48.531461 10677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:09:48.534958 10691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:48.536254 10677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:09:48.536418 10677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "522e935aa7f543c2b12e5840b3afeb8b"
format_stamp: "Formatted at 2026-06-20 07:09:48 on dist-test-slave-4n6h"
server_key: "eb0220770026b09453c5e2a2c9763271"
server_key_iv: "00388201126fa9cf1378304358d17275"
server_key_version: "encryptionkey@0"
I20260620 07:09:48.536628 10677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:48.552098 10677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:48.556427 10677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:48.556715 10677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:48.557639 10677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:48.558966 10677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:48.559088 10677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:48.559221 10677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:48.559314 10677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:48.591676 10677 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:43897
I20260620 07:09:48.591722 10803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:43897 every 8 connection(s)
I20260620 07:09:48.592521 10677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:09:48.593349 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 10677
I20260620 07:09:48.593545 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:09:48.593997 6357 external_mini_cluster.cc:1428] Setting key c1280a5d2a0c9abe79efc888e35c185b
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
Jun 20 07:09:48 dist-test-slave-4n6h krb5kdc[10439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939388, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:48.621974 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:48.595077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:57967 (local address 127.6.53.126:36445)
0620 07:09:48.595244 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:48.595250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:48.596530 (+ 1280us) server_negotiation.cc:407] Connection header received
0620 07:09:48.598322 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:48.598332 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:48.598421 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:48.598530 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:48.600016 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:48.601126 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:48.603065 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:48.603844 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:48.603976 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:48.616123 (+ 12147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:48.616128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:48.616131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:48.616164 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:48.618714 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:48.619499 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:48.619506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:48.619526 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:48.619689 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:48.620074 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:48.620080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:48.620084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:48.620463 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:48.620626 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:48.621265 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 07:09:48.621529 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:09:48.622661 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:48.595569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:36445 (local address 127.6.53.66:57967)
0620 07:09:48.596362 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:48.596406 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:48.598021 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:48.598727 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:48.598735 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:48.599292 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:48.599836 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:48.599864 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:48.601272 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:48.601279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:48.602927 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:48.602939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:48.604055 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:48.604060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:48.604242 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:48.612096 (+ 7854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:48.612135 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:48.615880 (+ 3745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:48.618857 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:48.618869 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:48.618889 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:48.619373 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:48.619804 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:48.619807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:48.619810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:48.619968 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:48.620590 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:48.620595 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:48.620908 (+ 313us) client_negotiation.cc:769] Sending connection context
0620 07:09:48.621136 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 07:09:48.621519 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":363,"threads_started":1}
I20260620 07:09:48.624157 10804 heartbeater.cc:347] Connected to a master server at 127.6.53.126:36445
I20260620 07:09:48.624456 10804 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:48.625085 10804 heartbeater.cc:511] Master 127.6.53.126:36445 requested a full tablet report, sending...
I20260620 07:09:48.626585 10483 ts_manager.cc:194] Registered new tserver with Master: 522e935aa7f543c2b12e5840b3afeb8b (127.6.53.66:43897)
I20260620 07:09:48.627422 10483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:57967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:09:48.637689 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:36445
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:41269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:48.953758 10809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:48.954073 10809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:48.954267 10809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:48.961279 10809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:48.961421 10809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:48.961618 10809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:09:48.971261 10809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:41269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:36445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:48.972509 10809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:48.973819 10809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:48.986127 10814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:48.986181 10815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:48.986348 10817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:48.987535 10809 server_base.cc:1034] running on GCE node
I20260620 07:09:48.988212 10809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:48.989269 10809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:48.990485 10809 hybrid_clock.cc:648] HybridClock initialized: now 1781939388990419 us; error 60 us; skew 500 ppm
I20260620 07:09:48.990799 10809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:48 dist-test-slave-4n6h krb5kdc[10439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939388, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:48.993752 10809 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:09:48.994902 10809 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:48.994999 10809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:48.995319 10809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:48.998106 10809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "fd885930810943ec8ff9c14d3a109963"
format_stamp: "Formatted at 2026-06-20 07:09:48 on dist-test-slave-4n6h"
server_key: "90b2c05a30c5ec59b88f5179d5d9f40a"
server_key_iv: "0a5c772aedde6c5ec46f10cfc4fdac7a"
server_key_version: "encryptionkey@0"
I20260620 07:09:48.998970 10809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "fd885930810943ec8ff9c14d3a109963"
format_stamp: "Formatted at 2026-06-20 07:09:48 on dist-test-slave-4n6h"
server_key: "90b2c05a30c5ec59b88f5179d5d9f40a"
server_key_iv: "0a5c772aedde6c5ec46f10cfc4fdac7a"
server_key_version: "encryptionkey@0"
I20260620 07:09:49.003890 10809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:09:49.007524 10823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:49.008771 10809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:09:49.008950 10809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "fd885930810943ec8ff9c14d3a109963"
format_stamp: "Formatted at 2026-06-20 07:09:48 on dist-test-slave-4n6h"
server_key: "90b2c05a30c5ec59b88f5179d5d9f40a"
server_key_iv: "0a5c772aedde6c5ec46f10cfc4fdac7a"
server_key_version: "encryptionkey@0"
I20260620 07:09:49.009186 10809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:49.027714 10809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:49.031540 10809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:49.031767 10809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:49.032639 10809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:49.033866 10809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:49.033949 10809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:49.034103 10809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:49.034184 10809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:49.066444 10809 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:37987
I20260620 07:09:49.066526 10935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:37987 every 8 connection(s)
I20260620 07:09:49.067306 10809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:09:49.075026 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 10809
I20260620 07:09:49.075213 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:09:49.075644 6357 external_mini_cluster.cc:1428] Setting key ba98ea701aefc67392a57b53fff3de20
Jun 20 07:09:49 dist-test-slave-4n6h krb5kdc[10439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939388, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:49.086880 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.069811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:33279 (local address 127.6.53.126:36445)
0620 07:09:49.069983 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:49.069987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:49.071160 (+ 1173us) server_negotiation.cc:407] Connection header received
0620 07:09:49.072952 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:49.072969 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:49.073056 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:49.073165 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:49.074715 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.075553 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.077457 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.077897 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.078041 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.081441 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:49.081446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:49.081449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:49.081483 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:49.083938 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:49.084673 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:49.084678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:49.084680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:49.084800 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:49.085215 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:49.085220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:49.085223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:49.085478 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:49.085635 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:49.086319 (+ 684us) server_negotiation.cc:299] Negotiation successful
0620 07:09:49.086529 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:09:49.087881 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.070308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:36445 (local address 127.6.53.67:33279)
0620 07:09:49.070997 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:49.071038 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:49.072682 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:49.073387 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:49.073395 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:49.073896 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:49.074535 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:49.074557 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.075707 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.075711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:49.077293 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:49.077311 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.078126 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.078135 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:49.078436 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.079112 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:49.079133 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:49.081231 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:49.084090 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:49.084099 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:49.084116 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:49.084536 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:49.084909 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:49.084915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:49.084919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:49.085099 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:49.085605 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:49.085613 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:49.085941 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 07:09:49.086197 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 07:09:49.086768 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":286,"threads_started":1}
I20260620 07:09:49.089362 10936 heartbeater.cc:347] Connected to a master server at 127.6.53.126:36445
I20260620 07:09:49.089716 10936 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:49.090621 10936 heartbeater.cc:511] Master 127.6.53.126:36445 requested a full tablet report, sending...
I20260620 07:09:49.092046 10483 ts_manager.cc:194] Registered new tserver with Master: fd885930810943ec8ff9c14d3a109963 (127.6.53.67:37987)
I20260620 07:09:49.092800 10483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:33279
I20260620 07:09:49.103111 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:09:49.121898 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.108402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58360 (local address 127.6.53.126:36445)
0620 07:09:49.108619 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:49.108628 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:49.109056 (+ 428us) server_negotiation.cc:407] Connection header received
0620 07:09:49.109190 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:49.109194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:49.109255 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:49.109332 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:49.110757 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.111456 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.112683 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.113160 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.113256 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.115551 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:49.115556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:49.115559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:49.115591 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:49.119169 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:49.119948 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:49.119970 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:49.119976 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:49.120115 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:49.120485 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:49.120493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:49.120497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:49.120810 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:49.121073 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:49.121363 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 07:09:49.121567 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:09:49.125939 10483 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 07:09:49.132557 10671 heartbeater.cc:503] Master 127.6.53.126:36445 was elected leader, sending a full tablet report...
W20260620 07:09:49.133632 10483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:09:49.156332 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.145147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:43897 (local address 127.0.0.1:36932)
0620 07:09:49.147673 (+ 2526us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:49.147691 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:49.147815 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:49.148430 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:49.148440 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:49.148475 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:49.148979 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:49.148996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.150988 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.150992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:49.153805 (+ 2813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:49.153819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.155470 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.155475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:49.155688 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.155715 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 07:09:49.155782 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 07:09:49.155883 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2395,"mutex_wait_us":170,"thread_start_us":1388,"threads_started":1}
I20260620 07:09:49.156441 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.145041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44044 (local address 127.6.53.67:37987)
0620 07:09:49.145888 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:49.145900 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:49.147142 (+ 1242us) server_negotiation.cc:407] Connection header received
0620 07:09:49.147529 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:49.147540 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:49.147795 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:49.148004 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:49.149262 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.150828 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.153967 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.155366 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.155501 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.155722 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:49.155938 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 07:09:49.156094 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":450,"threads_started":1}
I20260620 07:09:49.156474 10953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.148594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41646 (local address 127.6.53.65:33751)
0620 07:09:49.149266 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:49.149272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:49.149292 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:09:49.149361 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:49.149366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:49.149495 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:49.149673 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:49.150171 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.151671 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.153542 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.155024 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.155152 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.155367 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:49.156020 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 07:09:49.156173 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":322,"threads_started":1}
I20260620 07:09:49.156791 10951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.145722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.6.53.66:43897)
0620 07:09:49.146300 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:49.146311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:49.147772 (+ 1461us) server_negotiation.cc:407] Connection header received
0620 07:09:49.147955 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:49.147965 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:49.148209 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:49.148401 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:49.149192 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.150831 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.153991 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.155256 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.156113 (+ 857us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.156290 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:49.156351 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 07:09:49.156513 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":263,"threads_started":1}
I20260620 07:09:49.157123 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.144731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:37987 (local address 127.0.0.1:44044)
0620 07:09:49.146988 (+ 2257us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:49.147034 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:49.147245 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:49.148600 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:49.148604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:49.148625 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:49.149077 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:49.149090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.150987 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.150992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:49.153792 (+ 2800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:49.153806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.155561 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.155571 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:49.155740 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.155780 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 07:09:49.155897 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 07:09:49.156030 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1966,"mutex_wait_us":249,"thread_start_us":2558,"threads_started":1}
I20260620 07:09:49.157387 10952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.147661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:33751 (local address 127.0.0.1:41646)
0620 07:09:49.148307 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:49.148325 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:49.148418 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:49.149656 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:49.149661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:49.149685 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:49.150015 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:49.150024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.151815 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.151820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:49.153382 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:49.153400 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.155215 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.155221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:49.155442 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.155542 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 07:09:49.155666 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 07:09:49.155861 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":11,"thread_start_us":408,"threads_started":1}
I20260620 07:09:49.160254 10871 tablet_service.cc:1467] Processing CreateTablet for tablet 0f12244c958348b4ae64b0a5ea78a777 (DEFAULT_TABLE table=test-table [id=4f1c7c85d3d34923a5a3032babe20663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:49.160508 10739 tablet_service.cc:1467] Processing CreateTablet for tablet 0f12244c958348b4ae64b0a5ea78a777 (DEFAULT_TABLE table=test-table [id=4f1c7c85d3d34923a5a3032babe20663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:49.160509 10606 tablet_service.cc:1467] Processing CreateTablet for tablet 0f12244c958348b4ae64b0a5ea78a777 (DEFAULT_TABLE table=test-table [id=4f1c7c85d3d34923a5a3032babe20663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:49.161315 10871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f12244c958348b4ae64b0a5ea78a777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:49.161737 10739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f12244c958348b4ae64b0a5ea78a777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:49.161737 10606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f12244c958348b4ae64b0a5ea78a777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:49.172377 10954 tablet_bootstrap.cc:492] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963: Bootstrap starting.
I20260620 07:09:49.175880 10955 tablet_bootstrap.cc:492] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b: Bootstrap starting.
I20260620 07:09:49.176707 10954 tablet_bootstrap.cc:654] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:49.178059 10954 log.cc:824] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:49.178898 10956 tablet_bootstrap.cc:492] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d: Bootstrap starting.
I20260620 07:09:49.179493 10955 tablet_bootstrap.cc:654] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:49.180809 10955 log.cc:824] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:49.182407 10956 tablet_bootstrap.cc:654] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:49.183555 10956 log.cc:824] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:49.183755 10955 tablet_bootstrap.cc:492] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b: No bootstrap required, opened a new log
I20260620 07:09:49.184026 10955 ts_tablet_manager.cc:1397] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 07:09:49.184902 10954 tablet_bootstrap.cc:492] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963: No bootstrap required, opened a new log
I20260620 07:09:49.185140 10954 ts_tablet_manager.cc:1397] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20260620 07:09:49.186030 10956 tablet_bootstrap.cc:492] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d: No bootstrap required, opened a new log
I20260620 07:09:49.186316 10956 ts_tablet_manager.cc:1397] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260620 07:09:49.193732 10956 raft_consensus.cc:348] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd885930810943ec8ff9c14d3a109963" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37987 } } peers { permanent_uuid: "25adde3ed19c432d8afdb4d53a2c832d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33751 } } peers { permanent_uuid: "522e935aa7f543c2b12e5840b3afeb8b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43897 } }
I20260620 07:09:49.194082 10956 raft_consensus.cc:374] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:49.194319 10956 raft_consensus.cc:729] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25adde3ed19c432d8afdb4d53a2c832d, State: Initialized, Role: FOLLOWER
I20260620 07:09:49.194433 10955 raft_consensus.cc:348] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd885930810943ec8ff9c14d3a109963" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37987 } } peers { permanent_uuid: "25adde3ed19c432d8afdb4d53a2c832d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33751 } } peers { permanent_uuid: "522e935aa7f543c2b12e5840b3afeb8b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43897 } }
I20260620 07:09:49.194832 10955 raft_consensus.cc:374] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:49.194934 10955 raft_consensus.cc:729] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522e935aa7f543c2b12e5840b3afeb8b, State: Initialized, Role: FOLLOWER
I20260620 07:09:49.194980 10956 consensus_queue.cc:260] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd885930810943ec8ff9c14d3a109963" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37987 } } peers { permanent_uuid: "25adde3ed19c432d8afdb4d53a2c832d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33751 } } peers { permanent_uuid: "522e935aa7f543c2b12e5840b3afeb8b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43897 } }
I20260620 07:09:49.195494 10955 consensus_queue.cc:260] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd885930810943ec8ff9c14d3a109963" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37987 } } peers { permanent_uuid: "25adde3ed19c432d8afdb4d53a2c832d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33751 } } peers { permanent_uuid: "522e935aa7f543c2b12e5840b3afeb8b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43897 } }
I20260620 07:09:49.196822 10956 ts_tablet_manager.cc:1428] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s
I20260620 07:09:49.196928 10804 heartbeater.cc:503] Master 127.6.53.126:36445 was elected leader, sending a full tablet report...
I20260620 07:09:49.195870 10954 raft_consensus.cc:348] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd885930810943ec8ff9c14d3a109963" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37987 } } peers { permanent_uuid: "25adde3ed19c432d8afdb4d53a2c832d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33751 } } peers { permanent_uuid: "522e935aa7f543c2b12e5840b3afeb8b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43897 } }
I20260620 07:09:49.197377 10954 raft_consensus.cc:374] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:49.197479 10954 raft_consensus.cc:729] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd885930810943ec8ff9c14d3a109963, State: Initialized, Role: FOLLOWER
I20260620 07:09:49.198012 10954 consensus_queue.cc:260] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd885930810943ec8ff9c14d3a109963" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37987 } } peers { permanent_uuid: "25adde3ed19c432d8afdb4d53a2c832d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33751 } } peers { permanent_uuid: "522e935aa7f543c2b12e5840b3afeb8b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43897 } }
I20260620 07:09:49.199131 10936 heartbeater.cc:503] Master 127.6.53.126:36445 was elected leader, sending a full tablet report...
I20260620 07:09:49.199757 10954 ts_tablet_manager.cc:1428] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260620 07:09:49.201130 10955 ts_tablet_manager.cc:1428] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
W20260620 07:09:49.318768 10937 tablet.cc:2369] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:09:49.344408 10805 tablet.cc:2369] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:09:49.354224 10672 tablet.cc:2369] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:09:49.362685 10960 raft_consensus.cc:482] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:09:49.363010 10960 raft_consensus.cc:504] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd885930810943ec8ff9c14d3a109963" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37987 } } peers { permanent_uuid: "25adde3ed19c432d8afdb4d53a2c832d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33751 } } peers { permanent_uuid: "522e935aa7f543c2b12e5840b3afeb8b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43897 } }
I20260620 07:09:49.364756 10960 leader_election.cc:288] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd885930810943ec8ff9c14d3a109963 (127.6.53.67:37987), 522e935aa7f543c2b12e5840b3afeb8b (127.6.53.66:43897)
I20260620 07:09:49.371493 10951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.365271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:45173 (local address 127.6.53.66:43897)
0620 07:09:49.365504 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:49.365511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:49.366093 (+ 582us) server_negotiation.cc:407] Connection header received
0620 07:09:49.366325 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:49.366332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:49.366396 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:49.366494 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:49.367103 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.367893 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.369727 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.370566 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.370720 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.370815 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:49.371112 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 07:09:49.371229 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:09:49.371614 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.365301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:51197 (local address 127.6.53.67:37987)
0620 07:09:49.365504 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:49.365511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:49.366010 (+ 499us) server_negotiation.cc:407] Connection header received
0620 07:09:49.366236 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:49.366240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:49.366356 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:49.366451 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:09:49.367041 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.367844 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.369865 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.370712 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.370855 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.370972 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:49.371242 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 07:09:49.371359 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:09:49.371901 10964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.365241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:43897 (local address 127.6.53.65:45173)
0620 07:09:49.365990 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:49.366004 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:49.366159 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:49.366517 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:49.366521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:49.366545 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:49.366944 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:49.366953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.368026 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.368038 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:49.369578 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:49.369590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.370773 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.370779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:49.370944 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.370974 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:09:49.371090 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 07:09:49.371213 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":6,"thread_start_us":455,"threads_started":1}
I20260620 07:09:49.372079 10963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.365109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:37987 (local address 127.6.53.65:51197)
0620 07:09:49.365869 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:49.365888 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:49.366088 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:49.366471 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:49.366475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:49.366498 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:09:49.366896 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:49.366908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.367967 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.367976 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:49.369730 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:49.369742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.370930 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.370937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:49.371093 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.371116 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:09:49.371216 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 07:09:49.371339 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":86,"thread_start_us":399,"threads_started":1}
I20260620 07:09:49.372957 10891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f12244c958348b4ae64b0a5ea78a777" candidate_uuid: "25adde3ed19c432d8afdb4d53a2c832d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd885930810943ec8ff9c14d3a109963" is_pre_election: true
I20260620 07:09:49.372936 10759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f12244c958348b4ae64b0a5ea78a777" candidate_uuid: "25adde3ed19c432d8afdb4d53a2c832d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522e935aa7f543c2b12e5840b3afeb8b" is_pre_election: true
I20260620 07:09:49.373428 10891 raft_consensus.cc:2454] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25adde3ed19c432d8afdb4d53a2c832d in term 0.
I20260620 07:09:49.373425 10759 raft_consensus.cc:2454] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25adde3ed19c432d8afdb4d53a2c832d in term 0.
I20260620 07:09:49.374040 10560 leader_election.cc:302] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25adde3ed19c432d8afdb4d53a2c832d, 522e935aa7f543c2b12e5840b3afeb8b; no voters:
I20260620 07:09:49.374504 10960 raft_consensus.cc:2781] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:09:49.374644 10960 raft_consensus.cc:482] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:09:49.374754 10960 raft_consensus.cc:3037] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:49.376847 10960 raft_consensus.cc:504] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd885930810943ec8ff9c14d3a109963" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37987 } } peers { permanent_uuid: "25adde3ed19c432d8afdb4d53a2c832d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33751 } } peers { permanent_uuid: "522e935aa7f543c2b12e5840b3afeb8b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43897 } }
I20260620 07:09:49.377475 10960 leader_election.cc:288] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [CANDIDATE]: Term 1 election: Requested vote from peers fd885930810943ec8ff9c14d3a109963 (127.6.53.67:37987), 522e935aa7f543c2b12e5840b3afeb8b (127.6.53.66:43897)
I20260620 07:09:49.377897 10891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f12244c958348b4ae64b0a5ea78a777" candidate_uuid: "25adde3ed19c432d8afdb4d53a2c832d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd885930810943ec8ff9c14d3a109963"
I20260620 07:09:49.378003 10759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f12244c958348b4ae64b0a5ea78a777" candidate_uuid: "25adde3ed19c432d8afdb4d53a2c832d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522e935aa7f543c2b12e5840b3afeb8b"
I20260620 07:09:49.378123 10891 raft_consensus.cc:3037] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:49.378237 10759 raft_consensus.cc:3037] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:49.380313 10891 raft_consensus.cc:2454] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25adde3ed19c432d8afdb4d53a2c832d in term 1.
I20260620 07:09:49.380435 10759 raft_consensus.cc:2454] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25adde3ed19c432d8afdb4d53a2c832d in term 1.
I20260620 07:09:49.380708 10559 leader_election.cc:302] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25adde3ed19c432d8afdb4d53a2c832d, fd885930810943ec8ff9c14d3a109963; no voters:
I20260620 07:09:49.381083 10960 raft_consensus.cc:2781] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:49.381567 10960 raft_consensus.cc:686] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [term 1 LEADER]: Becoming Leader. State: Replica: 25adde3ed19c432d8afdb4d53a2c832d, State: Running, Role: LEADER
I20260620 07:09:49.382148 10960 consensus_queue.cc:237] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd885930810943ec8ff9c14d3a109963" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37987 } } peers { permanent_uuid: "25adde3ed19c432d8afdb4d53a2c832d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33751 } } peers { permanent_uuid: "522e935aa7f543c2b12e5840b3afeb8b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43897 } }
I20260620 07:09:49.386718 10481 catalog_manager.cc:5495] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d reported cstate change: term changed from 0 to 1, leader changed from <none> to 25adde3ed19c432d8afdb4d53a2c832d (127.6.53.65). New cstate: current_term: 1 leader_uuid: "25adde3ed19c432d8afdb4d53a2c832d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd885930810943ec8ff9c14d3a109963" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25adde3ed19c432d8afdb4d53a2c832d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e935aa7f543c2b12e5840b3afeb8b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43897 } health_report { overall_health: UNKNOWN } } }
I20260620 07:09:49.455624 10953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.450211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41662 (local address 127.6.53.65:33751)
0620 07:09:49.450459 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:49.450465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:49.450593 (+ 128us) server_negotiation.cc:407] Connection header received
0620 07:09:49.450835 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:49.450839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:49.450916 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:49.451099 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:09:49.451461 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.452185 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.453488 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.453896 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.454034 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.454553 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:09:49.455059 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:09:49.455232 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:49.455283 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 07:09:49.455361 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 07:09:49.463423 10891 raft_consensus.cc:1261] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963 [term 1 FOLLOWER]: Refusing update from remote peer 25adde3ed19c432d8afdb4d53a2c832d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:49.463511 10759 raft_consensus.cc:1261] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b [term 1 FOLLOWER]: Refusing update from remote peer 25adde3ed19c432d8afdb4d53a2c832d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:49.464509 10960 consensus_queue.cc:1035] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd885930810943ec8ff9c14d3a109963" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:49.464798 10965 consensus_queue.cc:1035] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e935aa7f543c2b12e5840b3afeb8b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:49.479157 10968 mvcc.cc:205] Tried to move back new op lower bound from 7298823739232440320 to 7298823738912911360. Current Snapshot: MvccSnapshot[applied={T|T < 7298823739232440320}]
I20260620 07:09:49.480400 10970 mvcc.cc:205] Tried to move back new op lower bound from 7298823739232440320 to 7298823738912911360. Current Snapshot: MvccSnapshot[applied={T|T < 7298823739232440320}]
I20260620 07:09:49.501585 10482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:09:49.501870 10482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:09:49.505359 10482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1200917b6a6c4463b4d589f04c14a996: Sending DeleteTablet for 3 replicas of tablet 0f12244c958348b4ae64b0a5ea78a777
I20260620 07:09:49.506862 10739 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f12244c958348b4ae64b0a5ea78a777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:49 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:36932
I20260620 07:09:49.506909 10871 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f12244c958348b4ae64b0a5ea78a777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:49 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:44044
I20260620 07:09:49.508056 10606 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f12244c958348b4ae64b0a5ea78a777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:49 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:41646
I20260620 07:09:49.508837 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 10544
I20260620 07:09:49.516963 10978 tablet_replica.cc:330] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963: stopping tablet replica
I20260620 07:09:49.517510 10978 raft_consensus.cc:2229] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:09:49.517629 10980 tablet_replica.cc:330] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d: stopping tablet replica
I20260620 07:09:49.518155 10978 raft_consensus.cc:2258] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:49.518121 10980 raft_consensus.cc:2229] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:09:49.518939 10980 raft_consensus.cc:2258] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:49.519347 10979 tablet_replica.cc:330] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b: stopping tablet replica
I20260620 07:09:49.519773 10979 raft_consensus.cc:2229] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:09:49.520284 10979 raft_consensus.cc:2258] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:49.520785 10978 ts_tablet_manager.cc:1905] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:49.520979 10980 ts_tablet_manager.cc:1905] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:49.522325 10979 ts_tablet_manager.cc:1905] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:49.528139 10980 ts_tablet_manager.cc:1918] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:09:49.528334 10980 log.cc:1196] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/0f12244c958348b4ae64b0a5ea78a777
I20260620 07:09:49.528932 10980 ts_tablet_manager.cc:1939] T 0f12244c958348b4ae64b0a5ea78a777 P 25adde3ed19c432d8afdb4d53a2c832d: Deleting consensus metadata
I20260620 07:09:49.529273 10979 ts_tablet_manager.cc:1918] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:09:49.529481 10979 log.cc:1196] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/0f12244c958348b4ae64b0a5ea78a777
I20260620 07:09:49.530018 10979 ts_tablet_manager.cc:1939] T 0f12244c958348b4ae64b0a5ea78a777 P 522e935aa7f543c2b12e5840b3afeb8b: Deleting consensus metadata
I20260620 07:09:49.530857 10467 catalog_manager.cc:4841] TS 25adde3ed19c432d8afdb4d53a2c832d (127.6.53.65:33751): tablet 0f12244c958348b4ae64b0a5ea78a777 (table test-table [id=4f1c7c85d3d34923a5a3032babe20663]) successfully deleted
I20260620 07:09:49.531503 10978 ts_tablet_manager.cc:1918] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:09:49.531706 10978 log.cc:1196] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/0f12244c958348b4ae64b0a5ea78a777
I20260620 07:09:49.531751 10468 catalog_manager.cc:4841] TS 522e935aa7f543c2b12e5840b3afeb8b (127.6.53.66:43897): tablet 0f12244c958348b4ae64b0a5ea78a777 (table test-table [id=4f1c7c85d3d34923a5a3032babe20663]) successfully deleted
I20260620 07:09:49.532191 10978 ts_tablet_manager.cc:1939] T 0f12244c958348b4ae64b0a5ea78a777 P fd885930810943ec8ff9c14d3a109963: Deleting consensus metadata
I20260620 07:09:49.533756 10467 catalog_manager.cc:4841] TS fd885930810943ec8ff9c14d3a109963 (127.6.53.67:37987): tablet 0f12244c958348b4ae64b0a5ea78a777 (table test-table [id=4f1c7c85d3d34923a5a3032babe20663]) successfully deleted
Jun 20 07:09:49 dist-test-slave-4n6h krb5kdc[10439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:09:49.550282 10953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.510045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.6.53.65:33751)
0620 07:09:49.510203 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:49.510209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:49.531987 (+ 21778us) server_negotiation.cc:407] Connection header received
0620 07:09:49.532137 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:49.532142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:49.532232 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:49.532325 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:49.534544 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.535743 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.537498 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.538171 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.538359 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.543465 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:49.543480 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:49.543499 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:49.543555 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:49.547549 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:49.548203 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:49.548212 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:49.548216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:49.548386 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:49.548721 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:49.548726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:49.548729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:49.549160 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:49.549440 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:49.549637 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 07:09:49.549875 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:09:49.551059 10666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:49.664063 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 10544
I20260620 07:09:49.682240 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 10677
Jun 20 07:09:49 dist-test-slave-4n6h krb5kdc[10439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:09:49.697027 10951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.683170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.6.53.66:43897)
0620 07:09:49.683342 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:49.683346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:49.683363 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:09:49.683442 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:49.683446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:49.683507 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:49.683620 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:49.684730 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.685435 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.686675 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.687046 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.687153 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.691213 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:49.691226 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:49.691241 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:49.691272 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:49.694506 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:49.695109 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:49.695116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:49.695118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:49.695236 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:49.695578 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:49.695581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:49.695583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:49.695940 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:49.696150 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:49.696386 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 07:09:49.696568 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:09:49.697849 10799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:49.785014 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 10677
I20260620 07:09:49.801595 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 10809
Jun 20 07:09:49 dist-test-slave-4n6h krb5kdc[10439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:09:49.817406 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:49.802404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.6.53.67:37987)
0620 07:09:49.802589 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:49.802597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:49.802626 (+ 29us) server_negotiation.cc:407] Connection header received
0620 07:09:49.802860 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:49.802866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:49.802954 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:49.803122 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:49.804482 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.805208 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.806694 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:49.807215 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:49.807372 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:49.811900 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:49.811915 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:49.811931 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:49.811964 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:49.814849 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:49.815445 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:49.815453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:49.815455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:49.815575 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:49.816098 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:49.816101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:49.816104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:49.816439 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:49.816653 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:49.816942 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 07:09:49.817094 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:09:49.818073 10931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:49.909185 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 10809
I20260620 07:09:49.925025 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 10452
I20260620 07:09:49.926121 10513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:50.033221 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 10452
2026-06-20T07:09:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4246 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T07:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:09:50Z Disabled control of system clock
I20260620 07:09:50.099344 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:40013
--webserver_interface=127.6.53.126
--webserver_port=0
--builtin_ntp_servers=127.6.53.84:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:40013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 07:09:50.409014 10989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:50.409291 10989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:50.409374 10989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:50.416448 10989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:50.416550 10989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:50.416589 10989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:50.416637 10989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:50.416673 10989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:50.425758 10989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:40013
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:40013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:50.427044 10989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:50.428777 10989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:50.440335 10994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:50.440428 10995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:50.440444 10997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:50.441305 10989 server_base.cc:1034] running on GCE node
I20260620 07:09:50.442062 10989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:50.443413 10989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:50.444640 10989 hybrid_clock.cc:648] HybridClock initialized: now 1781939390444591 us; error 59 us; skew 500 ppm
I20260620 07:09:50.445019 10989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:09:50.447376 10989 webserver.cc:460] Webserver started at http://127.6.53.126:33059/ using document root <none> and password file <none>
I20260620 07:09:50.448100 10989 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:50.448199 10989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:50.448526 10989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:50.451403 10989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "8d5c8a76f5a74c35b3ca5c65f27c5bc2"
format_stamp: "Formatted at 2026-06-20 07:09:50 on dist-test-slave-4n6h"
server_key: "283cec076975e59c17a238077912af61"
server_key_iv: "1f42a83caae0d5713bb8c3e3aef4ffc6"
server_key_version: "encryptionkey@0"
I20260620 07:09:50.452205 10989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "8d5c8a76f5a74c35b3ca5c65f27c5bc2"
format_stamp: "Formatted at 2026-06-20 07:09:50 on dist-test-slave-4n6h"
server_key: "283cec076975e59c17a238077912af61"
server_key_iv: "1f42a83caae0d5713bb8c3e3aef4ffc6"
server_key_version: "encryptionkey@0"
I20260620 07:09:50.457105 10989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:09:50.460388 11003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:50.461651 10989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 07:09:50.461826 10989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "8d5c8a76f5a74c35b3ca5c65f27c5bc2"
format_stamp: "Formatted at 2026-06-20 07:09:50 on dist-test-slave-4n6h"
server_key: "283cec076975e59c17a238077912af61"
server_key_iv: "1f42a83caae0d5713bb8c3e3aef4ffc6"
server_key_version: "encryptionkey@0"
I20260620 07:09:50.462044 10989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:50.467458 10989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:50.468176 10989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:50.468461 10989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:50.485332 10989 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:40013
I20260620 07:09:50.485363 11054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:40013 every 8 connection(s)
I20260620 07:09:50.486680 10989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:50.490125 11055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:50.495263 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 10989
I20260620 07:09:50.495463 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:09:50.496088 6357 external_mini_cluster.cc:1428] Setting key 0216c62d435fcfb63d88122d5338854b
I20260620 07:09:50.501516 11055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2: Bootstrap starting.
I20260620 07:09:50.505336 11055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:50.505380 11058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:50.498097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35016 (local address 127.6.53.126:40013)
0620 07:09:50.499252 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:50.499269 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:50.499319 (+ 50us) server_negotiation.cc:407] Connection header received
0620 07:09:50.501540 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:50.501575 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:50.502078 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:50.502486 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:50.503513 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:50.503521 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:50.503539 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:50.503561 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:50.503632 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:50.503654 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:50.503861 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:50.503978 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 07:09:50.504227 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":382,"threads_started":1}
I20260620 07:09:50.506449 11055 log.cc:824] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:50.509330 11055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2: No bootstrap required, opened a new log
I20260620 07:09:50.516722 11055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5c8a76f5a74c35b3ca5c65f27c5bc2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40013 } }
I20260620 07:09:50.517045 11055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:50.517139 11055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5c8a76f5a74c35b3ca5c65f27c5bc2, State: Initialized, Role: FOLLOWER
I20260620 07:09:50.517730 11055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5c8a76f5a74c35b3ca5c65f27c5bc2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40013 } }
I20260620 07:09:50.517956 11055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:09:50.518081 11055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:09:50.518255 11055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:50.520444 11055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5c8a76f5a74c35b3ca5c65f27c5bc2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40013 } }
I20260620 07:09:50.520925 11055 leader_election.cc:302] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d5c8a76f5a74c35b3ca5c65f27c5bc2; no voters:
I20260620 07:09:50.521519 11055 leader_election.cc:288] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:09:50.521656 11060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:50.522727 11060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [term 1 LEADER]: Becoming Leader. State: Replica: 8d5c8a76f5a74c35b3ca5c65f27c5bc2, State: Running, Role: LEADER
I20260620 07:09:50.523213 11060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5c8a76f5a74c35b3ca5c65f27c5bc2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40013 } }
I20260620 07:09:50.523586 11055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:50.526124 11062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d5c8a76f5a74c35b3ca5c65f27c5bc2. Latest consensus state: current_term: 1 leader_uuid: "8d5c8a76f5a74c35b3ca5c65f27c5bc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5c8a76f5a74c35b3ca5c65f27c5bc2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40013 } } }
I20260620 07:09:50.526588 11062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:50.526206 11061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d5c8a76f5a74c35b3ca5c65f27c5bc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d5c8a76f5a74c35b3ca5c65f27c5bc2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40013 } } }
I20260620 07:09:50.527211 11061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:50.530972 11069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:09:50.535379 11069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:09:50.543651 11069 catalog_manager.cc:1347] Generated new cluster ID: 72ebba028d5e4bc0b625049b7fbbf490
I20260620 07:09:50.543778 11069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:50.552160 11069 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:50.552590 11069 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:50.568841 11069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2: Generated new TSK 0
I20260620 07:09:50.569844 11069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:09:50.578032 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40013
--builtin_ntp_servers=127.6.53.84:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 07:09:50.896636 11079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:50.896916 11079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:50.897087 11079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:50.904160 11079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:50.904245 11079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:50.904444 11079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:09:50.913414 11079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:50.914669 11079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:50.916369 11079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:50.929095 11087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:50.929129 11084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:50.929056 11085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:50.930100 11079 server_base.cc:1034] running on GCE node
I20260620 07:09:50.930717 11079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:50.931598 11079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:50.932842 11079 hybrid_clock.cc:648] HybridClock initialized: now 1781939390932800 us; error 54 us; skew 500 ppm
I20260620 07:09:50.933141 11079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:09:50.935503 11079 webserver.cc:460] Webserver started at http://127.6.53.65:40283/ using document root <none> and password file <none>
I20260620 07:09:50.936161 11079 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:50.936280 11079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:50.936582 11079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:50.939381 11079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "a7e7bc6866dd4adc86e2b3ec8571711b"
format_stamp: "Formatted at 2026-06-20 07:09:50 on dist-test-slave-4n6h"
server_key: "13ec3afe6e02b1914084e09df99e4f37"
server_key_iv: "90b2f926e12f0e3c9051945e6fc3a5cf"
server_key_version: "encryptionkey@0"
I20260620 07:09:50.940136 11079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "a7e7bc6866dd4adc86e2b3ec8571711b"
format_stamp: "Formatted at 2026-06-20 07:09:50 on dist-test-slave-4n6h"
server_key: "13ec3afe6e02b1914084e09df99e4f37"
server_key_iv: "90b2f926e12f0e3c9051945e6fc3a5cf"
server_key_version: "encryptionkey@0"
I20260620 07:09:50.944805 11079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:09:50.948021 11093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:50.949297 11079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:09:50.949479 11079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "a7e7bc6866dd4adc86e2b3ec8571711b"
format_stamp: "Formatted at 2026-06-20 07:09:50 on dist-test-slave-4n6h"
server_key: "13ec3afe6e02b1914084e09df99e4f37"
server_key_iv: "90b2f926e12f0e3c9051945e6fc3a5cf"
server_key_version: "encryptionkey@0"
I20260620 07:09:50.949693 11079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:50.955283 11079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:50.956005 11079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:50.956256 11079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:50.957114 11079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:50.958423 11079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:50.958547 11079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:50.958671 11079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:50.958760 11079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:50.991058 11079 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:45457
I20260620 07:09:50.991138 11205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:45457 every 8 connection(s)
I20260620 07:09:50.992339 11079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:09:50.996150 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 11079
I20260620 07:09:50.996340 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:09:50.997017 6357 external_mini_cluster.cc:1428] Setting key 39c610d444289bbb6aaecab7d3b4651d
I20260620 07:09:50.999806 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40013
--builtin_ntp_servers=127.6.53.84:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 07:09:51.000728 11058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:50.994843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:49291 (local address 127.6.53.126:40013)
0620 07:09:50.995006 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:50.995010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:50.996405 (+ 1395us) server_negotiation.cc:407] Connection header received
0620 07:09:50.998230 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:50.998236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:50.998318 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:50.998435 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:50.999865 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:50.999871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:50.999877 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:50.999896 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:50.999933 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:50.999960 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:51.000089 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:51.000374 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 07:09:51.000433 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:09:51.001854 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:50.995338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:40013 (local address 127.6.53.65:49291)
0620 07:09:50.996206 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:50.996276 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:50.997987 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:50.998695 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:50.998706 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:50.998821 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:50.999511 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:09:50.999513 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:50.999546 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:09:50.999637 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:51.000151 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:51.000163 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:51.000174 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 07:09:51.000434 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 07:09:51.000680 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":300,"threads_started":1}
I20260620 07:09:51.003264 11206 heartbeater.cc:347] Connected to a master server at 127.6.53.126:40013
I20260620 07:09:51.003568 11206 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:51.004256 11206 heartbeater.cc:511] Master 127.6.53.126:40013 requested a full tablet report, sending...
I20260620 07:09:51.006662 11020 ts_manager.cc:194] Registered new tserver with Master: a7e7bc6866dd4adc86e2b3ec8571711b (127.6.53.65:45457)
W20260620 07:09:51.313573 11209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:51.313881 11209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:51.314040 11209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:51.322149 11209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:51.322301 11209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:51.322480 11209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:09:51.333711 11209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:51.335063 11209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:51.336815 11209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:51.349423 11217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:51.349483 11215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:51.349730 11214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:51.350539 11209 server_base.cc:1034] running on GCE node
I20260620 07:09:51.351080 11209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:51.352042 11209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:51.353233 11209 hybrid_clock.cc:648] HybridClock initialized: now 1781939391353176 us; error 58 us; skew 500 ppm
I20260620 07:09:51.353504 11209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:09:51.355798 11209 webserver.cc:460] Webserver started at http://127.6.53.66:45119/ using document root <none> and password file <none>
I20260620 07:09:51.356420 11209 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:51.356510 11209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:51.356782 11209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:51.359688 11209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "ff1e3c7d50cd43aa8aa085753ea0e40d"
format_stamp: "Formatted at 2026-06-20 07:09:51 on dist-test-slave-4n6h"
server_key: "5288426c4569c25a480c7eb84de0715f"
server_key_iv: "428d86eb5384132b46a8ac4b0f090eca"
server_key_version: "encryptionkey@0"
I20260620 07:09:51.360387 11209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "ff1e3c7d50cd43aa8aa085753ea0e40d"
format_stamp: "Formatted at 2026-06-20 07:09:51 on dist-test-slave-4n6h"
server_key: "5288426c4569c25a480c7eb84de0715f"
server_key_iv: "428d86eb5384132b46a8ac4b0f090eca"
server_key_version: "encryptionkey@0"
I20260620 07:09:51.365628 11209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:09:51.369020 11223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:51.370365 11209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:51.370517 11209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "ff1e3c7d50cd43aa8aa085753ea0e40d"
format_stamp: "Formatted at 2026-06-20 07:09:51 on dist-test-slave-4n6h"
server_key: "5288426c4569c25a480c7eb84de0715f"
server_key_iv: "428d86eb5384132b46a8ac4b0f090eca"
server_key_version: "encryptionkey@0"
I20260620 07:09:51.370744 11209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:51.376427 11209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:51.377213 11209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:51.377480 11209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:51.378464 11209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:51.379851 11209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:51.379938 11209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:51.380069 11209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:51.380122 11209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:51.414924 11209 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:38659
I20260620 07:09:51.415077 11335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:38659 every 8 connection(s)
I20260620 07:09:51.416272 11209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:09:51.419839 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 11209
I20260620 07:09:51.420120 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:09:51.420658 6357 external_mini_cluster.cc:1428] Setting key 78a268466f43e8706226549267ca5b75
I20260620 07:09:51.423888 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40013
--builtin_ntp_servers=127.6.53.84:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 07:09:51.425396 11058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:51.418961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:60739 (local address 127.6.53.126:40013)
0620 07:09:51.419167 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:51.419175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:51.420761 (+ 1586us) server_negotiation.cc:407] Connection header received
0620 07:09:51.422675 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:51.422681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:51.422746 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:51.422853 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:51.424345 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:51.424354 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:51.424362 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:51.424386 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:51.424433 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:51.424453 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:51.424599 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:51.424867 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 07:09:51.424944 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:09:51.425930 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:51.419535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:40013 (local address 127.6.53.66:60739)
0620 07:09:51.420618 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:51.420661 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:51.422357 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:51.423074 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:51.423082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:51.423170 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:51.423934 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:09:51.423936 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:51.423973 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:09:51.424120 (+ 147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:51.424553 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:51.424561 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:51.424566 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:09:51.424779 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 07:09:51.425031 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":340,"threads_started":1}
I20260620 07:09:51.427471 11336 heartbeater.cc:347] Connected to a master server at 127.6.53.126:40013
I20260620 07:09:51.427886 11336 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:51.428647 11336 heartbeater.cc:511] Master 127.6.53.126:40013 requested a full tablet report, sending...
I20260620 07:09:51.430377 11020 ts_manager.cc:194] Registered new tserver with Master: ff1e3c7d50cd43aa8aa085753ea0e40d (127.6.53.66:38659)
W20260620 07:09:51.766618 11339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:51.766929 11339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:51.767024 11339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:51.774379 11339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:51.774482 11339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:51.774626 11339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:09:51.784078 11339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:51.785375 11339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:51.787168 11339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:51.800525 11345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:51.800611 11347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:51.800650 11344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:51.801519 11339 server_base.cc:1034] running on GCE node
I20260620 07:09:51.802088 11339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:51.802999 11339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:51.804184 11339 hybrid_clock.cc:648] HybridClock initialized: now 1781939391804128 us; error 50 us; skew 500 ppm
I20260620 07:09:51.804481 11339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:09:51.806702 11339 webserver.cc:460] Webserver started at http://127.6.53.67:44985/ using document root <none> and password file <none>
I20260620 07:09:51.807340 11339 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:51.807451 11339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:51.807789 11339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:51.810680 11339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "804e03f64b19497b86124b6d9f824c0a"
format_stamp: "Formatted at 2026-06-20 07:09:51 on dist-test-slave-4n6h"
server_key: "855ea2196e1f1ef6924f245ae62e3c9e"
server_key_iv: "cb1265f3816104972066ea826810bb0c"
server_key_version: "encryptionkey@0"
I20260620 07:09:51.811546 11339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "804e03f64b19497b86124b6d9f824c0a"
format_stamp: "Formatted at 2026-06-20 07:09:51 on dist-test-slave-4n6h"
server_key: "855ea2196e1f1ef6924f245ae62e3c9e"
server_key_iv: "cb1265f3816104972066ea826810bb0c"
server_key_version: "encryptionkey@0"
I20260620 07:09:51.816613 11339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:09:51.819828 11353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:51.821071 11339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:09:51.821244 11339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "804e03f64b19497b86124b6d9f824c0a"
format_stamp: "Formatted at 2026-06-20 07:09:51 on dist-test-slave-4n6h"
server_key: "855ea2196e1f1ef6924f245ae62e3c9e"
server_key_iv: "cb1265f3816104972066ea826810bb0c"
server_key_version: "encryptionkey@0"
I20260620 07:09:51.821465 11339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:51.827275 11339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:51.828075 11339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:51.828336 11339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:51.829226 11339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:51.830588 11339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:51.830705 11339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:51.830837 11339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:51.830922 11339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:51.868351 11339 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:33787
I20260620 07:09:51.868491 11465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:33787 every 8 connection(s)
I20260620 07:09:51.869680 11339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:09:51.876767 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 11339
I20260620 07:09:51.876936 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:09:51.877516 6357 external_mini_cluster.cc:1428] Setting key af748833443534dcb8650e70cc0416b4
I20260620 07:09:51.878504 11058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:51.872497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:35617 (local address 127.6.53.126:40013)
0620 07:09:51.872695 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:51.872699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:51.874110 (+ 1411us) server_negotiation.cc:407] Connection header received
0620 07:09:51.876015 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:51.876020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:51.876077 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:51.876158 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:51.877680 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:51.877685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:51.877688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:51.877708 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:51.877740 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:51.877755 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:51.877856 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:51.878154 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 07:09:51.878224 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:09:51.879545 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:51.872980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:40013 (local address 127.6.53.67:35617)
0620 07:09:51.873947 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:51.873994 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:51.875738 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:51.876366 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:51.876377 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:51.876502 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:51.877277 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:09:51.877282 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:51.877332 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:09:51.877438 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:51.877858 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:51.877869 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:51.877878 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 07:09:51.878167 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 07:09:51.878441 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":279,"threads_started":1}
I20260620 07:09:51.880865 11466 heartbeater.cc:347] Connected to a master server at 127.6.53.126:40013
I20260620 07:09:51.881147 11466 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:51.881698 11466 heartbeater.cc:511] Master 127.6.53.126:40013 requested a full tablet report, sending...
I20260620 07:09:51.882982 11019 ts_manager.cc:194] Registered new tserver with Master: 804e03f64b19497b86124b6d9f824c0a (127.6.53.67:33787)
I20260620 07:09:51.892673 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:09:51.899430 11058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:51.896846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.6.53.126:40013)
0620 07:09:51.897012 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:51.897019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:51.897512 (+ 493us) server_negotiation.cc:407] Connection header received
0620 07:09:51.897706 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:51.897712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:51.897768 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:51.897863 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:51.898825 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:51.898832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:51.898836 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:51.898854 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:51.898891 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:51.898903 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:51.899021 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:51.899112 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 07:09:51.899168 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:09:51.903116 11019 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:09:51.909840 11019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:09:51.942047 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:51.922363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.6.53.67:33787)
0620 07:09:51.937463 (+ 15100us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:51.937473 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:51.937508 (+ 35us) server_negotiation.cc:407] Connection header received
0620 07:09:51.937593 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:51.937602 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:51.937840 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:51.938022 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:51.941080 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:51.941089 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:51.941103 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:51.941123 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:51.941184 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:51.941207 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:51.941401 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:51.941542 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 07:09:51.941644 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14917,"thread_start_us":14731,"threads_started":1}
I20260620 07:09:51.942093 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:51.921354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:33787 (local address 127.0.0.1:40534)
0620 07:09:51.936791 (+ 15437us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:51.936812 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:51.936892 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:51.939606 (+ 2714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:51.939616 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:51.939752 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:51.940718 (+ 966us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:09:51.940725 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:51.940786 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:09:51.940891 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:51.941335 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:51.941351 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:51.941361 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 07:09:51.941555 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 07:09:51.941644 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15308,"thread_start_us":17964,"threads_started":1}
I20260620 07:09:51.942766 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:51.921915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41536 (local address 127.6.53.66:38659)
0620 07:09:51.937199 (+ 15284us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:51.937208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:51.937237 (+ 29us) server_negotiation.cc:407] Connection header received
0620 07:09:51.937463 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:51.937471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:51.939075 (+ 1604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:51.939256 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:51.941659 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:51.941667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:51.941681 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:51.941711 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:51.941765 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:51.941783 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:51.941881 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:51.942349 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 07:09:51.942437 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15094,"thread_start_us":15336,"threads_started":1}
I20260620 07:09:51.943333 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:51.921055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:38659 (local address 127.0.0.1:41536)
0620 07:09:51.936338 (+ 15283us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:51.936392 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:51.936560 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:51.940608 (+ 4048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:51.940612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:51.940639 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:51.941410 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:09:51.941413 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:51.941431 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:09:51.941478 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:51.941996 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:51.942004 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:51.942011 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:09:51.942155 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 07:09:51.942201 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14885,"thread_start_us":15615,"threads_started":1}
I20260620 07:09:51.946738 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:51.937702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:45457 (local address 127.0.0.1:42294)
0620 07:09:51.941413 (+ 3711us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:51.941430 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:51.941538 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:51.942382 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:51.942388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:51.942413 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:51.943623 (+ 1210us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:09:51.943629 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:51.943645 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:09:51.943684 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:51.944169 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:51.944174 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:51.944179 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:09:51.944305 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 07:09:51.944352 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3292,"thread_start_us":3127,"threads_started":1}
I20260620 07:09:51.947055 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:51.937774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42294 (local address 127.6.53.65:45457)
0620 07:09:51.941167 (+ 3393us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:51.941175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:51.941652 (+ 477us) server_negotiation.cc:407] Connection header received
0620 07:09:51.941724 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:51.941731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:51.941934 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:51.942068 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:51.943868 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:51.943875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:51.943886 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:51.943905 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:51.943962 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:51.943981 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:51.944114 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:51.946657 (+ 2543us) server_negotiation.cc:299] Negotiation successful
0620 07:09:51.946749 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3239,"thread_start_us":2543,"threads_started":1}
I20260620 07:09:51.946583 11401 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbae178368b4d9f88f5984915ca7599 (DEFAULT_TABLE table=test-table [id=5de26cd32b9c43528454137b37d8ea68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:51.946981 11271 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbae178368b4d9f88f5984915ca7599 (DEFAULT_TABLE table=test-table [id=5de26cd32b9c43528454137b37d8ea68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:51.948086 11401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbae178368b4d9f88f5984915ca7599. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:51.948307 11271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbae178368b4d9f88f5984915ca7599. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:51.949928 11141 tablet_service.cc:1467] Processing CreateTablet for tablet 8cbae178368b4d9f88f5984915ca7599 (DEFAULT_TABLE table=test-table [id=5de26cd32b9c43528454137b37d8ea68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:09:51.951161 11141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cbae178368b4d9f88f5984915ca7599. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:51.961802 11484 tablet_bootstrap.cc:492] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b: Bootstrap starting.
I20260620 07:09:51.964843 11484 tablet_bootstrap.cc:654] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:51.965751 11485 tablet_bootstrap.cc:492] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a: Bootstrap starting.
I20260620 07:09:51.966411 11484 log.cc:824] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:51.966838 11486 tablet_bootstrap.cc:492] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d: Bootstrap starting.
I20260620 07:09:51.969609 11485 tablet_bootstrap.cc:654] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:51.970094 11486 tablet_bootstrap.cc:654] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:51.970850 11485 log.cc:824] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:51.971192 11484 tablet_bootstrap.cc:492] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b: No bootstrap required, opened a new log
I20260620 07:09:51.971210 11486 log.cc:824] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:51.971486 11484 ts_tablet_manager.cc:1397] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 07:09:51.973897 11485 tablet_bootstrap.cc:492] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a: No bootstrap required, opened a new log
I20260620 07:09:51.973909 11486 tablet_bootstrap.cc:492] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d: No bootstrap required, opened a new log
I20260620 07:09:51.974159 11486 ts_tablet_manager.cc:1397] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260620 07:09:51.974159 11485 ts_tablet_manager.cc:1397] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 07:09:51.979269 11484 raft_consensus.cc:348] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1e3c7d50cd43aa8aa085753ea0e40d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38659 } } peers { permanent_uuid: "804e03f64b19497b86124b6d9f824c0a" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 33787 } } peers { permanent_uuid: "a7e7bc6866dd4adc86e2b3ec8571711b" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45457 } }
I20260620 07:09:51.979621 11484 raft_consensus.cc:374] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:51.979723 11484 raft_consensus.cc:729] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7e7bc6866dd4adc86e2b3ec8571711b, State: Initialized, Role: FOLLOWER
I20260620 07:09:51.980322 11484 consensus_queue.cc:260] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1e3c7d50cd43aa8aa085753ea0e40d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38659 } } peers { permanent_uuid: "804e03f64b19497b86124b6d9f824c0a" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 33787 } } peers { permanent_uuid: "a7e7bc6866dd4adc86e2b3ec8571711b" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45457 } }
I20260620 07:09:51.981379 11206 heartbeater.cc:503] Master 127.6.53.126:40013 was elected leader, sending a full tablet report...
I20260620 07:09:51.982007 11484 ts_tablet_manager.cc:1428] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260620 07:09:51.984742 11485 raft_consensus.cc:348] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1e3c7d50cd43aa8aa085753ea0e40d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38659 } } peers { permanent_uuid: "804e03f64b19497b86124b6d9f824c0a" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 33787 } } peers { permanent_uuid: "a7e7bc6866dd4adc86e2b3ec8571711b" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45457 } }
I20260620 07:09:51.985160 11485 raft_consensus.cc:374] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:51.984949 11486 raft_consensus.cc:348] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1e3c7d50cd43aa8aa085753ea0e40d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38659 } } peers { permanent_uuid: "804e03f64b19497b86124b6d9f824c0a" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 33787 } } peers { permanent_uuid: "a7e7bc6866dd4adc86e2b3ec8571711b" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45457 } }
I20260620 07:09:51.985270 11485 raft_consensus.cc:729] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804e03f64b19497b86124b6d9f824c0a, State: Initialized, Role: FOLLOWER
I20260620 07:09:51.985338 11486 raft_consensus.cc:374] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:51.985426 11486 raft_consensus.cc:729] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1e3c7d50cd43aa8aa085753ea0e40d, State: Initialized, Role: FOLLOWER
I20260620 07:09:51.985993 11486 consensus_queue.cc:260] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1e3c7d50cd43aa8aa085753ea0e40d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38659 } } peers { permanent_uuid: "804e03f64b19497b86124b6d9f824c0a" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 33787 } } peers { permanent_uuid: "a7e7bc6866dd4adc86e2b3ec8571711b" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45457 } }
I20260620 07:09:51.985992 11485 consensus_queue.cc:260] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1e3c7d50cd43aa8aa085753ea0e40d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38659 } } peers { permanent_uuid: "804e03f64b19497b86124b6d9f824c0a" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 33787 } } peers { permanent_uuid: "a7e7bc6866dd4adc86e2b3ec8571711b" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45457 } }
I20260620 07:09:51.987663 11466 heartbeater.cc:503] Master 127.6.53.126:40013 was elected leader, sending a full tablet report...
I20260620 07:09:51.987717 11336 heartbeater.cc:503] Master 127.6.53.126:40013 was elected leader, sending a full tablet report...
I20260620 07:09:51.987897 11485 ts_tablet_manager.cc:1428] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260620 07:09:51.988092 11486 ts_tablet_manager.cc:1428] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
W20260620 07:09:51.994596 11207 tablet.cc:2369] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:09:52.121865 11467 tablet.cc:2369] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:09:52.168267 11337 tablet.cc:2369] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:09:52.228972 11490 raft_consensus.cc:482] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:09:52.229321 11490 raft_consensus.cc:504] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1e3c7d50cd43aa8aa085753ea0e40d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38659 } } peers { permanent_uuid: "804e03f64b19497b86124b6d9f824c0a" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 33787 } } peers { permanent_uuid: "a7e7bc6866dd4adc86e2b3ec8571711b" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45457 } }
I20260620 07:09:52.231055 11490 leader_election.cc:288] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff1e3c7d50cd43aa8aa085753ea0e40d (127.6.53.66:38659), 804e03f64b19497b86124b6d9f824c0a (127.6.53.67:33787)
I20260620 07:09:52.234674 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:52.231380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:43837 (local address 127.6.53.66:38659)
0620 07:09:52.231569 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:52.231574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:52.232570 (+ 996us) server_negotiation.cc:407] Connection header received
0620 07:09:52.232687 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:52.232697 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:52.232762 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:52.232891 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:52.233969 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:52.233975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:52.233980 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:52.233997 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:52.234035 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:52.234052 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:52.234188 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:52.234316 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 07:09:52.234376 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:09:52.234786 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:52.231686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:49559 (local address 127.6.53.67:33787)
0620 07:09:52.231927 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:52.231936 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:52.232767 (+ 831us) server_negotiation.cc:407] Connection header received
0620 07:09:52.232942 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:52.232947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:52.233023 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:52.233136 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:52.234170 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:52.234176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:52.234180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:52.234197 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:52.234234 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:52.234250 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:52.234372 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:52.234479 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 07:09:52.234534 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:09:52.235038 11494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:52.231503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:33787 (local address 127.6.53.65:49559)
0620 07:09:52.232659 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:52.232674 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:52.232773 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:52.233164 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:52.233167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:52.233184 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:52.233945 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:09:52.233948 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:52.233962 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:09:52.234000 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:52.234360 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:52.234364 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:52.234369 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:09:52.234522 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 07:09:52.234578 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"thread_start_us":802,"threads_started":1}
I20260620 07:09:52.235241 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:52.231451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:38659 (local address 127.6.53.65:43837)
0620 07:09:52.232431 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:52.232449 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:52.232586 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:52.232948 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:52.232952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:52.232976 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:52.233739 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:09:52.233744 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:52.233764 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:09:52.233814 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:52.234175 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:52.234180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:52.234185 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:09:52.234354 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 07:09:52.234407 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":474,"threads_started":1}
I20260620 07:09:52.236038 11421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbae178368b4d9f88f5984915ca7599" candidate_uuid: "a7e7bc6866dd4adc86e2b3ec8571711b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804e03f64b19497b86124b6d9f824c0a" is_pre_election: true
I20260620 07:09:52.236137 11291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbae178368b4d9f88f5984915ca7599" candidate_uuid: "a7e7bc6866dd4adc86e2b3ec8571711b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff1e3c7d50cd43aa8aa085753ea0e40d" is_pre_election: true
I20260620 07:09:52.236524 11421 raft_consensus.cc:2454] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7e7bc6866dd4adc86e2b3ec8571711b in term 0.
I20260620 07:09:52.236667 11291 raft_consensus.cc:2454] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7e7bc6866dd4adc86e2b3ec8571711b in term 0.
I20260620 07:09:52.237201 11097 leader_election.cc:302] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 804e03f64b19497b86124b6d9f824c0a, a7e7bc6866dd4adc86e2b3ec8571711b; no voters:
I20260620 07:09:52.237661 11490 raft_consensus.cc:2781] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:09:52.237809 11490 raft_consensus.cc:482] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:09:52.237900 11490 raft_consensus.cc:3037] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:52.239998 11490 raft_consensus.cc:504] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1e3c7d50cd43aa8aa085753ea0e40d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38659 } } peers { permanent_uuid: "804e03f64b19497b86124b6d9f824c0a" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 33787 } } peers { permanent_uuid: "a7e7bc6866dd4adc86e2b3ec8571711b" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45457 } }
I20260620 07:09:52.240563 11490 leader_election.cc:288] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [CANDIDATE]: Term 1 election: Requested vote from peers ff1e3c7d50cd43aa8aa085753ea0e40d (127.6.53.66:38659), 804e03f64b19497b86124b6d9f824c0a (127.6.53.67:33787)
I20260620 07:09:52.240897 11291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbae178368b4d9f88f5984915ca7599" candidate_uuid: "a7e7bc6866dd4adc86e2b3ec8571711b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff1e3c7d50cd43aa8aa085753ea0e40d"
I20260620 07:09:52.241078 11291 raft_consensus.cc:3037] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:52.241077 11421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cbae178368b4d9f88f5984915ca7599" candidate_uuid: "a7e7bc6866dd4adc86e2b3ec8571711b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804e03f64b19497b86124b6d9f824c0a"
I20260620 07:09:52.241263 11421 raft_consensus.cc:3037] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:52.243316 11291 raft_consensus.cc:2454] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7e7bc6866dd4adc86e2b3ec8571711b in term 1.
I20260620 07:09:52.243571 11421 raft_consensus.cc:2454] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7e7bc6866dd4adc86e2b3ec8571711b in term 1.
I20260620 07:09:52.243741 11096 leader_election.cc:302] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7e7bc6866dd4adc86e2b3ec8571711b, ff1e3c7d50cd43aa8aa085753ea0e40d; no voters:
I20260620 07:09:52.244045 11490 raft_consensus.cc:2781] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:52.244441 11490 raft_consensus.cc:686] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [term 1 LEADER]: Becoming Leader. State: Replica: a7e7bc6866dd4adc86e2b3ec8571711b, State: Running, Role: LEADER
I20260620 07:09:52.244891 11490 consensus_queue.cc:237] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1e3c7d50cd43aa8aa085753ea0e40d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38659 } } peers { permanent_uuid: "804e03f64b19497b86124b6d9f824c0a" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 33787 } } peers { permanent_uuid: "a7e7bc6866dd4adc86e2b3ec8571711b" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45457 } }
I20260620 07:09:52.249418 11019 catalog_manager.cc:5495] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b reported cstate change: term changed from 0 to 1, leader changed from <none> to a7e7bc6866dd4adc86e2b3ec8571711b (127.6.53.65). New cstate: current_term: 1 leader_uuid: "a7e7bc6866dd4adc86e2b3ec8571711b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1e3c7d50cd43aa8aa085753ea0e40d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "804e03f64b19497b86124b6d9f824c0a" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 33787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7e7bc6866dd4adc86e2b3ec8571711b" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 45457 } health_report { overall_health: HEALTHY } } }
I20260620 07:09:52.292219 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:52.290291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.6.53.65:45457)
0620 07:09:52.290483 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:52.290488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:52.290505 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:09:52.290675 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:52.290679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:52.290754 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:52.290833 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:52.291677 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:52.291681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:52.291684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:52.291695 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:52.291719 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:52.291729 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:52.291810 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:52.291864 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 07:09:52.291905 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:09:52.300446 11291 raft_consensus.cc:1261] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d [term 1 FOLLOWER]: Refusing update from remote peer a7e7bc6866dd4adc86e2b3ec8571711b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:52.300433 11421 raft_consensus.cc:1261] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a [term 1 FOLLOWER]: Refusing update from remote peer a7e7bc6866dd4adc86e2b3ec8571711b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:09:52.301343 11495 consensus_queue.cc:1035] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff1e3c7d50cd43aa8aa085753ea0e40d" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:52.301693 11490 consensus_queue.cc:1035] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [LEADER]: Connected to new peer: Peer: permanent_uuid: "804e03f64b19497b86124b6d9f824c0a" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 33787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:09:52.312491 11498 mvcc.cc:205] Tried to move back new op lower bound from 7298823750853140480 to 7298823750638362624. Current Snapshot: MvccSnapshot[applied={T|T < 7298823750853140480}]
I20260620 07:09:52.313401 11500 mvcc.cc:205] Tried to move back new op lower bound from 7298823750853140480 to 7298823750638362624. Current Snapshot: MvccSnapshot[applied={T|T < 7298823750853140480}]
I20260620 07:09:52.328219 11020 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:09:52.328545 11020 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:09:52.332515 11020 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d5c8a76f5a74c35b3ca5c65f27c5bc2: Sending DeleteTablet for 3 replicas of tablet 8cbae178368b4d9f88f5984915ca7599
I20260620 07:09:52.333523 11271 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cbae178368b4d9f88f5984915ca7599 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:52 UTC) from {username='slave'} at 127.0.0.1:41536
I20260620 07:09:52.333636 11141 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cbae178368b4d9f88f5984915ca7599 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:52 UTC) from {username='slave'} at 127.0.0.1:42294
I20260620 07:09:52.333698 11401 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cbae178368b4d9f88f5984915ca7599 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:09:52 UTC) from {username='slave'} at 127.0.0.1:40534
I20260620 07:09:52.337797 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 11079
I20260620 07:09:52.346421 11508 tablet_replica.cc:330] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d: stopping tablet replica
I20260620 07:09:52.346927 11509 tablet_replica.cc:330] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b: stopping tablet replica
I20260620 07:09:52.346930 11508 raft_consensus.cc:2229] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:09:52.347394 11509 raft_consensus.cc:2229] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:09:52.347481 11508 raft_consensus.cc:2258] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:52.348073 11509 raft_consensus.cc:2258] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:52.349779 11508 ts_tablet_manager.cc:1905] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:52.349959 11509 ts_tablet_manager.cc:1905] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:52.356971 11508 ts_tablet_manager.cc:1918] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:09:52.357026 11509 ts_tablet_manager.cc:1918] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:09:52.357182 11509 log.cc:1196] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/8cbae178368b4d9f88f5984915ca7599
I20260620 07:09:52.357210 11508 log.cc:1196] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/8cbae178368b4d9f88f5984915ca7599
I20260620 07:09:52.357774 11509 ts_tablet_manager.cc:1939] T 8cbae178368b4d9f88f5984915ca7599 P a7e7bc6866dd4adc86e2b3ec8571711b: Deleting consensus metadata
I20260620 07:09:52.357767 11508 ts_tablet_manager.cc:1939] T 8cbae178368b4d9f88f5984915ca7599 P ff1e3c7d50cd43aa8aa085753ea0e40d: Deleting consensus metadata
I20260620 07:09:52.359546 11006 catalog_manager.cc:4841] TS ff1e3c7d50cd43aa8aa085753ea0e40d (127.6.53.66:38659): tablet 8cbae178368b4d9f88f5984915ca7599 (table test-table [id=5de26cd32b9c43528454137b37d8ea68]) successfully deleted
I20260620 07:09:52.359885 11006 catalog_manager.cc:4841] TS a7e7bc6866dd4adc86e2b3ec8571711b (127.6.53.65:45457): tablet 8cbae178368b4d9f88f5984915ca7599 (table test-table [id=5de26cd32b9c43528454137b37d8ea68]) successfully deleted
I20260620 07:09:52.361286 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:52.339238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42310 (local address 127.6.53.65:45457)
0620 07:09:52.339425 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:52.339433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:52.358532 (+ 19099us) server_negotiation.cc:407] Connection header received
0620 07:09:52.359116 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:52.359122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:52.359190 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:52.359268 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:52.360639 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:52.360647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:52.360656 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:52.360671 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:52.360698 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:52.360709 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:52.360843 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:52.360964 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 07:09:52.361020 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:09:52.361464 11510 tablet_replica.cc:330] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a: stopping tablet replica
I20260620 07:09:52.361979 11201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:52.361948 11510 raft_consensus.cc:2229] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:09:52.362505 11510 raft_consensus.cc:2258] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:09:52.364919 11510 ts_tablet_manager.cc:1905] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:09:52.370846 11510 ts_tablet_manager.cc:1918] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:09:52.371037 11510 log.cc:1196] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/8cbae178368b4d9f88f5984915ca7599
I20260620 07:09:52.371490 11510 ts_tablet_manager.cc:1939] T 8cbae178368b4d9f88f5984915ca7599 P 804e03f64b19497b86124b6d9f824c0a: Deleting consensus metadata
I20260620 07:09:52.373142 11007 catalog_manager.cc:4841] TS 804e03f64b19497b86124b6d9f824c0a (127.6.53.67:33787): tablet 8cbae178368b4d9f88f5984915ca7599 (table test-table [id=5de26cd32b9c43528454137b37d8ea68]) successfully deleted
I20260620 07:09:52.460706 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 11079
I20260620 07:09:52.478963 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 11209
I20260620 07:09:52.481910 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:52.479759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41546 (local address 127.6.53.66:38659)
0620 07:09:52.479941 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:52.479949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:52.479978 (+ 29us) server_negotiation.cc:407] Connection header received
0620 07:09:52.480182 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:52.480186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:52.480270 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:52.480380 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:52.481274 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:52.481279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:52.481284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:52.481301 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:52.481338 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:52.481349 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:52.481471 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:52.481558 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 07:09:52.481624 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:09:52.482501 11331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:52.569038 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 11209
I20260620 07:09:52.586004 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 11339
I20260620 07:09:52.588851 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:52.586886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.6.53.67:33787)
0620 07:09:52.587103 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:52.587108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:52.587130 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:09:52.587211 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:52.587220 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:52.587324 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:52.587414 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:52.588189 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:52.588193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:52.588197 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:52.588212 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:52.588246 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:52.588259 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:52.588374 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:52.588422 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 07:09:52.588469 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 07:09:52.589489 11461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:52.675981 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 11339
I20260620 07:09:52.692000 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 10989
I20260620 07:09:52.693104 11050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:52.796416 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 10989
2026-06-20T07:09:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2761 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 07:09:52.836836 6357 mini_oidc.cc:152] Starting JWKS server
I20260620 07:09:52.839345 6357 webserver.cc:460] Webserver started at https://127.0.0.1:33933/ using document root <none> and password file <none>
I20260620 07:09:52.839658 6357 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 07:09:52.840255 6357 webserver.cc:460] Webserver started at http://127.0.0.1:38319/ using document root <none> and password file <none>
2026-06-20T07:09:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:09:52Z Disabled control of system clock
I20260620 07:09:52.865767 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:41401
--webserver_interface=127.6.53.126
--webserver_port=0
--builtin_ntp_servers=127.6.53.84:43449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:41401
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33933/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/testchainca.cert with env {}
W20260620 07:09:53.174307 11521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:53.174569 11521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:53.174750 11521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:53.181710 11521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:53.181833 11521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:53.181906 11521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:53.182008 11521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:53.182088 11521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 07:09:53.182180 11521 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 07:09:53.182253 11521 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33933/jwks.json
I20260620 07:09:53.191838 11521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:43449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:41401
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:41401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33933/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:53.193032 11521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:53.194301 11521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:53.206035 11526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:53.206060 11527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:53.206172 11529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:53.206838 11521 server_base.cc:1034] running on GCE node
I20260620 07:09:53.207656 11521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:53.208979 11521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:53.210170 11521 hybrid_clock.cc:648] HybridClock initialized: now 1781939393210129 us; error 30 us; skew 500 ppm
I20260620 07:09:53.210523 11521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:09:53.212568 11521 webserver.cc:460] Webserver started at http://127.6.53.126:32801/ using document root <none> and password file <none>
I20260620 07:09:53.213176 11521 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:53.213261 11521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:53.213627 11521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:53.216472 11521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "4901fe424b16461eb31dc17605f28bf9"
format_stamp: "Formatted at 2026-06-20 07:09:53 on dist-test-slave-4n6h"
server_key: "cb85d0780e614e5a95bd406df0ea56ab"
server_key_iv: "1b0c94494b8df1420dc2f95c51ce12c8"
server_key_version: "encryptionkey@0"
I20260620 07:09:53.217187 11521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "4901fe424b16461eb31dc17605f28bf9"
format_stamp: "Formatted at 2026-06-20 07:09:53 on dist-test-slave-4n6h"
server_key: "cb85d0780e614e5a95bd406df0ea56ab"
server_key_iv: "1b0c94494b8df1420dc2f95c51ce12c8"
server_key_version: "encryptionkey@0"
I20260620 07:09:53.221973 11521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 07:09:53.225304 11535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:53.226661 11521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:09:53.226845 11521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "4901fe424b16461eb31dc17605f28bf9"
format_stamp: "Formatted at 2026-06-20 07:09:53 on dist-test-slave-4n6h"
server_key: "cb85d0780e614e5a95bd406df0ea56ab"
server_key_iv: "1b0c94494b8df1420dc2f95c51ce12c8"
server_key_version: "encryptionkey@0"
I20260620 07:09:53.227051 11521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:53.250167 11521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:53.250989 11521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:53.251274 11521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:53.268901 11521 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:41401
I20260620 07:09:53.268921 11586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:41401 every 8 connection(s)
I20260620 07:09:53.270174 11521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:53.271937 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 11521
I20260620 07:09:53.272168 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:09:53.272776 6357 external_mini_cluster.cc:1428] Setting key e1affa52244b6470bf976a47dac07c81
I20260620 07:09:53.274118 11587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:53.287566 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:53.274625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44522 (local address 127.6.53.126:41401)
0620 07:09:53.275623 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:53.275638 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:53.275686 (+ 48us) server_negotiation.cc:407] Connection header received
0620 07:09:53.277398 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:53.277425 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:53.277981 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:53.278447 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:53.278958 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:53.280974 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:53.282665 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:53.283626 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:53.283811 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:53.284895 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:53.284905 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:53.284922 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:53.285647 (+ 725us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:53.285733 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:53.285758 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:53.285879 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:53.286046 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 07:09:53.286392 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":300,"threads_started":1}
I20260620 07:09:53.288635 11587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9: Bootstrap starting.
I20260620 07:09:53.291141 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:09:53.292059 11587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:53.293216 11587 log.cc:824] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:53.296078 11587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9: No bootstrap required, opened a new log
I20260620 07:09:53.303174 11587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4901fe424b16461eb31dc17605f28bf9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41401 } }
I20260620 07:09:53.303493 11587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:53.303566 11587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4901fe424b16461eb31dc17605f28bf9, State: Initialized, Role: FOLLOWER
I20260620 07:09:53.304013 11587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4901fe424b16461eb31dc17605f28bf9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41401 } }
I20260620 07:09:53.304200 11587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:09:53.304294 11587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:09:53.304414 11587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:53.306459 11587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4901fe424b16461eb31dc17605f28bf9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41401 } }
I20260620 07:09:53.306888 11587 leader_election.cc:302] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4901fe424b16461eb31dc17605f28bf9; no voters:
I20260620 07:09:53.307570 11587 leader_election.cc:288] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:09:53.307741 11592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:53.308447 11592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [term 1 LEADER]: Becoming Leader. State: Replica: 4901fe424b16461eb31dc17605f28bf9, State: Running, Role: LEADER
I20260620 07:09:53.308952 11592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4901fe424b16461eb31dc17605f28bf9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41401 } }
I20260620 07:09:53.309484 11587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:53.311604 11593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4901fe424b16461eb31dc17605f28bf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4901fe424b16461eb31dc17605f28bf9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41401 } } }
I20260620 07:09:53.311928 11593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:53.312701 11599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:09:53.313033 11594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4901fe424b16461eb31dc17605f28bf9. Latest consensus state: current_term: 1 leader_uuid: "4901fe424b16461eb31dc17605f28bf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4901fe424b16461eb31dc17605f28bf9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41401 } } }
I20260620 07:09:53.313496 11594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:53.320640 11599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:09:53.328691 11599 catalog_manager.cc:1347] Generated new cluster ID: aa7f5e9551194f2bbc6ee14a7b49ab1c
I20260620 07:09:53.328791 11599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:53.346153 11599 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:53.347420 11599 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:53.354744 11599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4901fe424b16461eb31dc17605f28bf9: Generated new TSK 0
I20260620 07:09:53.355531 11599 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 07:09:53.394354 11617 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 07:09:53.394523 11617 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 07:09:53.395745 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:53.374177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.6.53.126:41401)
0620 07:09:53.374441 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:53.374448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:53.374765 (+ 317us) server_negotiation.cc:407] Connection header received
0620 07:09:53.375057 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:53.375064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:53.375160 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:53.375334 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
0620 07:09:53.375788 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:53.376972 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:53.378464 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:53.379190 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:53.379343 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:53.379807 (+ 464us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 07:09:53.394968 (+ 15161us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33933/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 07:09:53.395353 (+ 385us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44532: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33933/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":97}
W20260620 07:09:53.395874 11590 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44532: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33933/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 07:09:53.396955 11616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:53.374010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41401 (local address 127.0.0.1:44532)
0620 07:09:53.374663 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:53.374678 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:53.374815 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:53.375346 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:53.375349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:53.375358 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 07:09:53.375645 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:53.375652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:53.377139 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:53.377142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:53.378323 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:53.378335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:53.379361 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:53.379366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:53.379480 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:53.379533 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 07:09:53.395245 (+ 15712us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33933/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 07:09:53.395492 (+ 247us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.53.126:41401: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33933/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":341,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.53.126:41401: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33933/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 07:09:53.398924 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 11521
I20260620 07:09:53.400406 11582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:53.492871 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 11521
2026-06-20T07:09:53Z chronyd exiting
I20260620 07:09:53.542584 6357 test_util.cc:175] -----------------------------------------------
I20260620 07:09:53.542718 6357 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939352012798-6357-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (710 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 07:09:53.546341 6357 mini_oidc.cc:152] Starting JWKS server
I20260620 07:09:53.548098 6357 webserver.cc:460] Webserver started at https://127.0.0.1:39825/ using document root <none> and password file <none>
I20260620 07:09:53.548271 6357 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 07:09:53.548827 6357 webserver.cc:460] Webserver started at http://127.0.0.1:36593/ using document root <none> and password file <none>
2026-06-20T07:09:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:09:53Z Disabled control of system clock
I20260620 07:09:53.574168 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:46735
--webserver_interface=127.6.53.126
--webserver_port=0
--builtin_ntp_servers=127.6.53.84:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:46735
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39825/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/testchainca.cert with env {}
W20260620 07:09:53.906940 11624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:53.907212 11624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:53.907305 11624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:53.914455 11624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:53.914561 11624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:53.914597 11624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:53.914654 11624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:53.914696 11624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 07:09:53.914757 11624 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 07:09:53.914798 11624 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39825/jwks.json
I20260620 07:09:53.924981 11624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:46735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:46735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39825/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:53.926203 11624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:53.927438 11624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:53.939078 11630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:53.939107 11629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:53.939160 11632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:53.939695 11624 server_base.cc:1034] running on GCE node
I20260620 07:09:53.940480 11624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:53.941854 11624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:53.943118 11624 hybrid_clock.cc:648] HybridClock initialized: now 1781939393943073 us; error 38 us; skew 500 ppm
I20260620 07:09:53.943470 11624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:09:53.945750 11624 webserver.cc:460] Webserver started at http://127.6.53.126:40989/ using document root <none> and password file <none>
I20260620 07:09:53.946497 11624 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:53.946614 11624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:53.946945 11624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:53.949733 11624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "824b83287f3c4ed98ddab84458e55212"
format_stamp: "Formatted at 2026-06-20 07:09:53 on dist-test-slave-4n6h"
server_key: "75c93bb243ec364ff703622bc04cdc9a"
server_key_iv: "8773331d3193acc8138c10693b8a19b8"
server_key_version: "encryptionkey@0"
I20260620 07:09:53.950716 11624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "824b83287f3c4ed98ddab84458e55212"
format_stamp: "Formatted at 2026-06-20 07:09:53 on dist-test-slave-4n6h"
server_key: "75c93bb243ec364ff703622bc04cdc9a"
server_key_iv: "8773331d3193acc8138c10693b8a19b8"
server_key_version: "encryptionkey@0"
I20260620 07:09:53.957553 11624 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 07:09:53.962289 11638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:53.963958 11624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:53.964178 11624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "824b83287f3c4ed98ddab84458e55212"
format_stamp: "Formatted at 2026-06-20 07:09:53 on dist-test-slave-4n6h"
server_key: "75c93bb243ec364ff703622bc04cdc9a"
server_key_iv: "8773331d3193acc8138c10693b8a19b8"
server_key_version: "encryptionkey@0"
I20260620 07:09:53.964497 11624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:53.982436 11624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:53.983233 11624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:53.983523 11624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:54.002548 11624 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:46735
I20260620 07:09:54.002570 11689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:46735 every 8 connection(s)
I20260620 07:09:54.004102 11624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:54.009063 11690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:54.011304 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 11624
I20260620 07:09:54.011513 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:09:54.012202 6357 external_mini_cluster.cc:1428] Setting key 5fe3119869c61c65dd294801ea66f6b0
I20260620 07:09:54.025259 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:54.014055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.6.53.126:46735)
0620 07:09:54.015187 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:54.015198 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:54.015240 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:09:54.016956 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:54.016978 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:54.017467 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:54.017858 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:54.018448 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:54.019670 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:54.020940 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:54.021583 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:54.021784 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:54.022970 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:54.022978 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:54.022990 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:54.023521 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:54.023599 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:54.023619 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:54.023761 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:54.023966 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 07:09:54.024385 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":457,"threads_started":1}
I20260620 07:09:54.026566 11690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212: Bootstrap starting.
I20260620 07:09:54.028963 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:09:54.029865 11690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:54.030886 11690 log.cc:824] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:54.033622 11690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212: No bootstrap required, opened a new log
I20260620 07:09:54.040582 11690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824b83287f3c4ed98ddab84458e55212" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46735 } }
I20260620 07:09:54.041105 11690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:54.041227 11690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824b83287f3c4ed98ddab84458e55212, State: Initialized, Role: FOLLOWER
I20260620 07:09:54.041764 11690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824b83287f3c4ed98ddab84458e55212" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46735 } }
I20260620 07:09:54.041982 11690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:09:54.042140 11690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:09:54.042318 11690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:54.044369 11690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824b83287f3c4ed98ddab84458e55212" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46735 } }
I20260620 07:09:54.044871 11690 leader_election.cc:302] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 824b83287f3c4ed98ddab84458e55212; no voters:
I20260620 07:09:54.045449 11690 leader_election.cc:288] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:09:54.045603 11695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:54.046299 11695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [term 1 LEADER]: Becoming Leader. State: Replica: 824b83287f3c4ed98ddab84458e55212, State: Running, Role: LEADER
I20260620 07:09:54.046813 11695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824b83287f3c4ed98ddab84458e55212" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46735 } }
I20260620 07:09:54.047410 11690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:54.049444 11697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 824b83287f3c4ed98ddab84458e55212. Latest consensus state: current_term: 1 leader_uuid: "824b83287f3c4ed98ddab84458e55212" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824b83287f3c4ed98ddab84458e55212" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46735 } } }
I20260620 07:09:54.049845 11697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:54.050377 11696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "824b83287f3c4ed98ddab84458e55212" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824b83287f3c4ed98ddab84458e55212" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46735 } } }
I20260620 07:09:54.050660 11696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:54.051354 11703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:09:54.056655 11703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:09:54.065056 11703 catalog_manager.cc:1347] Generated new cluster ID: 37031040e2344c378ebe7e6a1f5499df
I20260620 07:09:54.065168 11703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:54.113277 11703 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:54.114704 11703 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:54.126899 11703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 824b83287f3c4ed98ddab84458e55212: Generated new TSK 0
I20260620 07:09:54.127940 11703 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 07:09:54.167491 11720 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 07:09:54.167622 11720 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260620 07:09:54.168828 11719 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:54.148621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:46735 (local address 127.0.0.1:59182)
0620 07:09:54.149333 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:54.149351 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:54.149459 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:54.149944 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:54.149947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:54.149955 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 07:09:54.150217 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:54.150224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:54.151714 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:54.151718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:54.152955 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:54.152967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:54.153984 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:54.153988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:54.154121 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:54.154169 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 07:09:54.168371 (+ 14202us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39825/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 07:09:54.168552 (+ 181us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.53.126:46735: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39825/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":453,"threads_started":1}
I20260620 07:09:54.168821 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:54.148629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.6.53.126:46735)
0620 07:09:54.148811 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:54.148817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:54.149429 (+ 612us) server_negotiation.cc:407] Connection header received
0620 07:09:54.149675 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:54.149680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:54.149769 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:54.149940 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0620 07:09:54.150419 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:54.151588 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:54.153118 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:54.153833 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:54.153954 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:54.154426 (+ 472us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 07:09:54.168158 (+ 13732us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39825/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 07:09:54.168510 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59182: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39825/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 07:09:54.168960 11693 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59182: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39825/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 07:09:54.170421 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 11624
I20260620 07:09:54.171470 11685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:54.264554 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 11624
2026-06-20T07:09:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (773 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 07:09:54.319743 6357 mini_oidc.cc:152] Starting JWKS server
I20260620 07:09:54.321511 6357 webserver.cc:460] Webserver started at https://127.0.0.1:36551/ using document root <none> and password file <none>
I20260620 07:09:54.321712 6357 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 07:09:54.322325 6357 webserver.cc:460] Webserver started at http://127.0.0.1:36491/ using document root <none> and password file <none>
2026-06-20T07:09:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:09:54Z Disabled control of system clock
I20260620 07:09:54.346817 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:38921
--webserver_interface=127.6.53.126
--webserver_port=0
--builtin_ntp_servers=127.6.53.84:35415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:38921
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36551/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 07:09:54.662609 11727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:54.662900 11727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:54.662985 11727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:54.670634 11727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:54.670734 11727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:54.670771 11727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:54.670828 11727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:54.670871 11727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 07:09:54.670930 11727 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 07:09:54.670969 11727 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36551/jwks.json
I20260620 07:09:54.679868 11727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:35415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:38921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:38921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36551/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:54.681039 11727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:54.682849 11727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:54.695220 11732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:54.695328 11735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:54.695358 11733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:54.696111 11727 server_base.cc:1034] running on GCE node
I20260620 07:09:54.696890 11727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:54.698288 11727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:54.699527 11727 hybrid_clock.cc:648] HybridClock initialized: now 1781939394699476 us; error 68 us; skew 500 ppm
I20260620 07:09:54.699870 11727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:09:54.702243 11727 webserver.cc:460] Webserver started at http://127.6.53.126:42759/ using document root <none> and password file <none>
I20260620 07:09:54.702955 11727 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:54.703075 11727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:54.703433 11727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:54.706203 11727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "a650a48e3a8341999ea3de43c11ab056"
format_stamp: "Formatted at 2026-06-20 07:09:54 on dist-test-slave-4n6h"
server_key: "48eb3258d02cce8f1a42f48e335eb9a7"
server_key_iv: "964cc1b7f919f9b403718e3afcd88166"
server_key_version: "encryptionkey@0"
I20260620 07:09:54.707044 11727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "a650a48e3a8341999ea3de43c11ab056"
format_stamp: "Formatted at 2026-06-20 07:09:54 on dist-test-slave-4n6h"
server_key: "48eb3258d02cce8f1a42f48e335eb9a7"
server_key_iv: "964cc1b7f919f9b403718e3afcd88166"
server_key_version: "encryptionkey@0"
I20260620 07:09:54.711834 11727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:09:54.715123 11741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:54.716377 11727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:09:54.716553 11727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "a650a48e3a8341999ea3de43c11ab056"
format_stamp: "Formatted at 2026-06-20 07:09:54 on dist-test-slave-4n6h"
server_key: "48eb3258d02cce8f1a42f48e335eb9a7"
server_key_iv: "964cc1b7f919f9b403718e3afcd88166"
server_key_version: "encryptionkey@0"
I20260620 07:09:54.716759 11727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:54.732899 11727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:54.733654 11727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:54.733934 11727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:54.751127 11727 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:38921
I20260620 07:09:54.751151 11792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:38921 every 8 connection(s)
I20260620 07:09:54.752420 11727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:54.753187 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 11727
I20260620 07:09:54.753415 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:09:54.754132 6357 external_mini_cluster.cc:1428] Setting key 62c11872fa06e4a53068dea41974938d
I20260620 07:09:54.756753 11793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:54.769209 11796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:54.756020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45606 (local address 127.6.53.126:38921)
0620 07:09:54.757063 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:54.757078 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:54.757127 (+ 49us) server_negotiation.cc:407] Connection header received
0620 07:09:54.759356 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:54.759382 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:54.759982 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:54.760426 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:54.760954 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:54.762983 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:54.764221 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:54.765192 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:54.765366 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:54.766445 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:54.766454 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:54.766473 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:54.767196 (+ 723us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:54.767287 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:54.767313 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:54.767445 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:54.767624 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 07:09:54.767962 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":325,"threads_started":1}
I20260620 07:09:54.770444 11793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056: Bootstrap starting.
I20260620 07:09:54.772814 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:09:54.773612 11793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:54.774623 11793 log.cc:824] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:54.777403 11793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056: No bootstrap required, opened a new log
I20260620 07:09:54.784752 11793 raft_consensus.cc:348] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a650a48e3a8341999ea3de43c11ab056" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38921 } }
I20260620 07:09:54.785087 11793 raft_consensus.cc:374] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:54.785200 11793 raft_consensus.cc:729] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a650a48e3a8341999ea3de43c11ab056, State: Initialized, Role: FOLLOWER
I20260620 07:09:54.785836 11793 consensus_queue.cc:260] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a650a48e3a8341999ea3de43c11ab056" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38921 } }
I20260620 07:09:54.786131 11793 raft_consensus.cc:388] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:09:54.786296 11793 raft_consensus.cc:482] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:09:54.786477 11793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:54.788489 11793 raft_consensus.cc:504] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a650a48e3a8341999ea3de43c11ab056" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38921 } }
I20260620 07:09:54.788923 11793 leader_election.cc:302] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a650a48e3a8341999ea3de43c11ab056; no voters:
I20260620 07:09:54.789429 11793 leader_election.cc:288] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:09:54.789592 11798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:54.790302 11798 raft_consensus.cc:686] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [term 1 LEADER]: Becoming Leader. State: Replica: a650a48e3a8341999ea3de43c11ab056, State: Running, Role: LEADER
I20260620 07:09:54.790853 11798 consensus_queue.cc:237] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a650a48e3a8341999ea3de43c11ab056" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38921 } }
I20260620 07:09:54.791639 11793 sys_catalog.cc:564] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:54.793866 11799 sys_catalog.cc:455] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a650a48e3a8341999ea3de43c11ab056" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a650a48e3a8341999ea3de43c11ab056" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38921 } } }
I20260620 07:09:54.794441 11799 sys_catalog.cc:458] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:54.794003 11800 sys_catalog.cc:455] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a650a48e3a8341999ea3de43c11ab056. Latest consensus state: current_term: 1 leader_uuid: "a650a48e3a8341999ea3de43c11ab056" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a650a48e3a8341999ea3de43c11ab056" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38921 } } }
I20260620 07:09:54.794761 11800 sys_catalog.cc:458] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:54.795800 11807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:09:54.800113 11807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:09:54.807953 11807 catalog_manager.cc:1347] Generated new cluster ID: fe76e799eb984afa825ce0c9df024ad6
I20260620 07:09:54.808068 11807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:54.819591 11807 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:54.821342 11807 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:54.828330 11807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a650a48e3a8341999ea3de43c11ab056: Generated new TSK 0
I20260620 07:09:54.829277 11807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:09:54.855515 11796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:54.847218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45610 (local address 127.6.53.126:38921)
0620 07:09:54.847402 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:54.847409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:54.847739 (+ 330us) server_negotiation.cc:407] Connection header received
0620 07:09:54.847941 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:54.847947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:54.848036 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:54.848216 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0620 07:09:54.848655 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:54.849762 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:54.851174 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:54.851870 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:54.852010 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:54.852502 (+ 492us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 07:09:54.854871 (+ 2369us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36551/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 07:09:54.855211 (+ 340us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45610: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36551/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 07:09:54.855674 11796 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45610: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36551/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 07:09:54.857246 11822 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:54.847042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:38921 (local address 127.0.0.1:45610)
0620 07:09:54.847629 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:54.847644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:54.847754 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:54.848258 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:54.848262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:54.848271 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 07:09:54.848513 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:54.848520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:54.849911 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:54.849915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:54.851027 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:54.851039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:54.852071 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:54.852076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:54.852208 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:54.852239 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 07:09:54.855073 (+ 2834us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36551/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 07:09:54.856821 (+ 1748us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.53.126:38921: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36551/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":1436,"thread_start_us":324,"threads_started":1}
I20260620 07:09:54.858230 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 11727
I20260620 07:09:54.859366 11788 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 07:09:54.867313 11823 webserver.cc:533] Webserver: sslize failed:
E20260620 07:09:54.867425 11823 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 07:09:54.947672 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 11727
2026-06-20T07:09:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (682 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T07:09:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:09:55Z Disabled control of system clock
I20260620 07:09:55.028126 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:46241
--webserver_interface=127.6.53.126
--webserver_port=0
--builtin_ntp_servers=127.6.53.84:34157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:46241
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/testchainca.cert with env {}
W20260620 07:09:55.338418 11828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:55.338724 11828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:55.338814 11828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:55.345927 11828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:55.346020 11828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:55.346058 11828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:55.346114 11828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:55.346150 11828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 07:09:55.346210 11828 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 07:09:55.346266 11828 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 07:09:55.355898 11828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:46241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:46241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:55.357129 11828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:55.358371 11828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:55.369609 11836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:55.369637 11834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:55.370133 11833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:55.370352 11828 server_base.cc:1034] running on GCE node
I20260620 07:09:55.371145 11828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:55.372483 11828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:55.373708 11828 hybrid_clock.cc:648] HybridClock initialized: now 1781939395373676 us; error 59 us; skew 500 ppm
I20260620 07:09:55.374059 11828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:09:55.376308 11828 webserver.cc:460] Webserver started at http://127.6.53.126:35001/ using document root <none> and password file <none>
I20260620 07:09:55.376986 11828 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:55.377116 11828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:55.377424 11828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:55.380188 11828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "9ddc7fd9711b4afdb30af1d5b837f0d5"
format_stamp: "Formatted at 2026-06-20 07:09:55 on dist-test-slave-4n6h"
server_key: "68453d4dd34cb33ecf5bbb7130cd833f"
server_key_iv: "c8eaf61412f7107c3782b56fb1b555b1"
server_key_version: "encryptionkey@0"
I20260620 07:09:55.380985 11828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "9ddc7fd9711b4afdb30af1d5b837f0d5"
format_stamp: "Formatted at 2026-06-20 07:09:55 on dist-test-slave-4n6h"
server_key: "68453d4dd34cb33ecf5bbb7130cd833f"
server_key_iv: "c8eaf61412f7107c3782b56fb1b555b1"
server_key_version: "encryptionkey@0"
I20260620 07:09:55.385850 11828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:09:55.389184 11842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:55.390532 11828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:55.390710 11828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "9ddc7fd9711b4afdb30af1d5b837f0d5"
format_stamp: "Formatted at 2026-06-20 07:09:55 on dist-test-slave-4n6h"
server_key: "68453d4dd34cb33ecf5bbb7130cd833f"
server_key_iv: "c8eaf61412f7107c3782b56fb1b555b1"
server_key_version: "encryptionkey@0"
I20260620 07:09:55.390914 11828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:55.411924 11828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:55.412760 11828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:55.413054 11828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:55.431015 11828 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:46241
I20260620 07:09:55.431032 11893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:46241 every 8 connection(s)
I20260620 07:09:55.432333 11828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:55.433993 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 11828
I20260620 07:09:55.434186 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:09:55.434842 6357 external_mini_cluster.cc:1428] Setting key 426f1767f9669914e571915b1ae7a915
I20260620 07:09:55.436360 11894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:55.449689 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:55.436693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51100 (local address 127.6.53.126:46241)
0620 07:09:55.437718 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:55.437733 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:55.437785 (+ 52us) server_negotiation.cc:407] Connection header received
0620 07:09:55.439629 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:55.439658 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:55.440265 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:55.440702 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:55.441211 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:55.443242 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:55.444517 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:55.445519 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:55.445702 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:55.446853 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:55.446863 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:55.446881 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:09:55.447623 (+ 742us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:55.447712 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:09:55.447739 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:55.447865 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:55.448070 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 07:09:55.448435 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":331,"threads_started":1}
I20260620 07:09:55.450788 11894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5: Bootstrap starting.
I20260620 07:09:55.453871 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:09:55.454530 11894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:55.455734 11894 log.cc:824] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:55.458665 11894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5: No bootstrap required, opened a new log
I20260620 07:09:55.465919 11894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ddc7fd9711b4afdb30af1d5b837f0d5" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46241 } }
I20260620 07:09:55.466235 11894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:55.466375 11894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ddc7fd9711b4afdb30af1d5b837f0d5, State: Initialized, Role: FOLLOWER
I20260620 07:09:55.466908 11894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ddc7fd9711b4afdb30af1d5b837f0d5" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46241 } }
I20260620 07:09:55.467106 11894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:09:55.467199 11894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:09:55.467322 11894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:55.469571 11894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ddc7fd9711b4afdb30af1d5b837f0d5" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46241 } }
I20260620 07:09:55.470008 11894 leader_election.cc:302] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ddc7fd9711b4afdb30af1d5b837f0d5; no voters:
I20260620 07:09:55.470566 11894 leader_election.cc:288] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:09:55.470736 11899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:55.471431 11899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9ddc7fd9711b4afdb30af1d5b837f0d5, State: Running, Role: LEADER
I20260620 07:09:55.471971 11899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ddc7fd9711b4afdb30af1d5b837f0d5" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46241 } }
I20260620 07:09:55.472627 11894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:55.474789 11901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ddc7fd9711b4afdb30af1d5b837f0d5. Latest consensus state: current_term: 1 leader_uuid: "9ddc7fd9711b4afdb30af1d5b837f0d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ddc7fd9711b4afdb30af1d5b837f0d5" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46241 } } }
I20260620 07:09:55.475199 11901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:55.475148 11900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ddc7fd9711b4afdb30af1d5b837f0d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ddc7fd9711b4afdb30af1d5b837f0d5" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 46241 } } }
I20260620 07:09:55.475464 11900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:55.476640 11905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:09:55.481041 11905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:09:55.489724 11905 catalog_manager.cc:1347] Generated new cluster ID: 406c0a14eb9c49d6a3738bdab838df63
I20260620 07:09:55.489823 11905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:55.506384 11905 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:55.507606 11905 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:55.515668 11905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ddc7fd9711b4afdb30af1d5b837f0d5: Generated new TSK 0
I20260620 07:09:55.516456 11905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:09:55.575640 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:55.541679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.6.53.126:46241)
0620 07:09:55.541910 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:55.541919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:55.542479 (+ 560us) server_negotiation.cc:407] Connection header received
0620 07:09:55.542718 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:55.542722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:55.542789 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:55.542919 (+ 130us) server_negotiation.cc:226] Negotiated authn=JWT
0620 07:09:55.543580 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:55.544312 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:55.546497 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:55.546948 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:55.547049 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:55.547666 (+ 617us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 07:09:55.575086 (+ 27420us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 07:09:55.575407 (+ 321us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51104: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 07:09:55.575800 11897 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51104: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 07:09:55.576490 11923 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:55.541630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:46241 (local address 127.0.0.1:51104)
0620 07:09:55.542339 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:55.542362 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:55.542536 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:55.542973 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:55.542980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:55.542992 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 07:09:55.543384 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:55.543398 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:55.544472 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:55.544480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:55.546339 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:55.546357 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:55.547130 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:55.547138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:55.547348 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:55.547398 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 07:09:55.575373 (+ 27975us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 07:09:55.575708 (+ 335us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.53.126:46241: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":365,"threads_started":1}
I20260620 07:09:55.578337 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 11828
I20260620 07:09:55.579591 11889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:55.673290 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 11828
2026-06-20T07:09:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (704 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (397 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (401 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:09:56 dist-test-slave-4n6h krb5kdc[11938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:09:56 dist-test-slave-4n6h krb5kdc[11938](info): set up 2 sockets
Jun 20 07:09:56 dist-test-slave-4n6h krb5kdc[11938](info): commencing operation
krb5kdc: starting...
W20260620 07:09:57.550835 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:09:57 dist-test-slave-4n6h krb5kdc[11938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:09:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:09:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:09:57.745725 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:42149
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:42149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:58.063951 11951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:58.064222 11951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:58.064369 11951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:58.071537 11951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:09:58.071657 11951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:58.071774 11951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:58.071870 11951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:09:58.071960 11951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:09:58.082140 11951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:42149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:42149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:58.083459 11951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:58.084725 11951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:58.096036 11956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:58.096163 11957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:58.096230 11959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:58.097573 11951 server_base.cc:1034] running on GCE node
I20260620 07:09:58.098441 11951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:58.099792 11951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:58.101060 11951 hybrid_clock.cc:648] HybridClock initialized: now 1781939398101028 us; error 92 us; skew 500 ppm
I20260620 07:09:58.101428 11951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:58 dist-test-slave-4n6h krb5kdc[11938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939398, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:58.104539 11951 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:09:58.106053 11951 webserver.cc:460] Webserver started at http://127.6.53.126:36051/ using document root <none> and password file <none>
I20260620 07:09:58.106807 11951 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:58.106951 11951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:58.107266 11951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:58.110102 11951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "30dc3c15489341c6a462330412366282"
format_stamp: "Formatted at 2026-06-20 07:09:58 on dist-test-slave-4n6h"
server_key: "188f557f67b71d5b1c56f8476a00331a"
server_key_iv: "6711d382744bcb441618c9dcabe06708"
server_key_version: "encryptionkey@0"
I20260620 07:09:58.110929 11951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "30dc3c15489341c6a462330412366282"
format_stamp: "Formatted at 2026-06-20 07:09:58 on dist-test-slave-4n6h"
server_key: "188f557f67b71d5b1c56f8476a00331a"
server_key_iv: "6711d382744bcb441618c9dcabe06708"
server_key_version: "encryptionkey@0"
I20260620 07:09:58.115851 11951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:09:58.119076 11966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:58.120311 11951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:58.120496 11951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "30dc3c15489341c6a462330412366282"
format_stamp: "Formatted at 2026-06-20 07:09:58 on dist-test-slave-4n6h"
server_key: "188f557f67b71d5b1c56f8476a00331a"
server_key_iv: "6711d382744bcb441618c9dcabe06708"
server_key_version: "encryptionkey@0"
I20260620 07:09:58.120702 11951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:58.135507 11951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:58.139653 11951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:58.140043 11951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:58.160019 11951 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:42149
I20260620 07:09:58.160037 12017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:42149 every 8 connection(s)
I20260620 07:09:58.161504 11951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:09:58.165273 12018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:09:58.166496 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 11951
I20260620 07:09:58.166721 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:09:58.167531 6357 external_mini_cluster.cc:1428] Setting key 32a57f554d9d3771367cd26d402a1930
I20260620 07:09:58.179991 12018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282: Bootstrap starting.
Jun 20 07:09:58 dist-test-slave-4n6h krb5kdc[11938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:58.183514 12018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282: Neither blocks nor log segments found. Creating new log.
I20260620 07:09:58.184969 12018 log.cc:824] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282: Log is configured to *not* fsync() on all Append() calls
I20260620 07:09:58.188180 12018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282: No bootstrap required, opened a new log
I20260620 07:09:58.190987 12021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:58.169633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.6.53.126:42149)
0620 07:09:58.170474 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:58.170486 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:58.170528 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:09:58.172170 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:58.172194 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:58.172580 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:58.172916 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:58.174585 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:58.175838 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:58.177633 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:58.178342 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:58.178466 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:58.182988 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:58.182996 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:58.183011 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:58.183049 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:58.186641 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:58.187282 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:58.187293 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:58.187303 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:58.187497 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:58.187845 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:58.187851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:58.187854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:58.188512 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:58.188808 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:58.189219 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 07:09:58.189619 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":395,"threads_started":1}
I20260620 07:09:58.197260 12018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc3c15489341c6a462330412366282" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42149 } }
I20260620 07:09:58.197594 12018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:09:58.197669 12018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30dc3c15489341c6a462330412366282, State: Initialized, Role: FOLLOWER
I20260620 07:09:58.198192 12018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc3c15489341c6a462330412366282" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42149 } }
I20260620 07:09:58.198446 12018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:09:58.198541 12018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:09:58.198673 12018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:09:58.200642 12018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc3c15489341c6a462330412366282" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42149 } }
I20260620 07:09:58.201062 12018 leader_election.cc:302] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30dc3c15489341c6a462330412366282; no voters:
I20260620 07:09:58.201584 12018 leader_election.cc:288] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:09:58.201686 12023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:09:58.203315 12023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [term 1 LEADER]: Becoming Leader. State: Replica: 30dc3c15489341c6a462330412366282, State: Running, Role: LEADER
I20260620 07:09:58.203788 12023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc3c15489341c6a462330412366282" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42149 } }
I20260620 07:09:58.203895 12018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:09:58.206092 12025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30dc3c15489341c6a462330412366282. Latest consensus state: current_term: 1 leader_uuid: "30dc3c15489341c6a462330412366282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc3c15489341c6a462330412366282" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42149 } } }
I20260620 07:09:58.206477 12025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:58.206535 12024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30dc3c15489341c6a462330412366282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30dc3c15489341c6a462330412366282" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42149 } } }
I20260620 07:09:58.206817 12024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282 [sys.catalog]: This master's current role is: LEADER
I20260620 07:09:58.211091 12030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:09:58.216099 12030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:09:58.224812 12030 catalog_manager.cc:1347] Generated new cluster ID: 3b1b6b0a5ea34dc6934bb5f95f8aed73
I20260620 07:09:58.224908 12030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:09:58.237854 12030 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:09:58.239178 12030 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:09:58.245878 12030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30dc3c15489341c6a462330412366282: Generated new TSK 0
I20260620 07:09:58.246840 12030 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:09:58.293017 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42149
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:58.632544 12044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:58.632807 12044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:58.632898 12044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:58.640027 12044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:58.640136 12044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:58.640246 12044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:09:58.649748 12044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:58.651100 12044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:58.652329 12044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:58.665128 12052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:58.665164 12049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:58.665220 12050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:58.666131 12044 server_base.cc:1034] running on GCE node
I20260620 07:09:58.666781 12044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:58.667708 12044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:58.668910 12044 hybrid_clock.cc:648] HybridClock initialized: now 1781939398668854 us; error 48 us; skew 500 ppm
I20260620 07:09:58.669229 12044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:58 dist-test-slave-4n6h krb5kdc[11938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939398, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:58.676046 12044 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:09:58.677495 12044 webserver.cc:460] Webserver started at http://127.6.53.65:43639/ using document root <none> and password file <none>
I20260620 07:09:58.678169 12044 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:58.678339 12044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:58.678642 12044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:58.681495 12044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "ae579a30c3be4a848169035c059b5cca"
format_stamp: "Formatted at 2026-06-20 07:09:58 on dist-test-slave-4n6h"
server_key: "fe9cc7a86d803b589c3f68199f5f41f9"
server_key_iv: "85c55fcbb56fa828ae36341758dad769"
server_key_version: "encryptionkey@0"
I20260620 07:09:58.682337 12044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "ae579a30c3be4a848169035c059b5cca"
format_stamp: "Formatted at 2026-06-20 07:09:58 on dist-test-slave-4n6h"
server_key: "fe9cc7a86d803b589c3f68199f5f41f9"
server_key_iv: "85c55fcbb56fa828ae36341758dad769"
server_key_version: "encryptionkey@0"
I20260620 07:09:58.687410 12044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 07:09:58.691000 12059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:58.692430 12044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:09:58.692592 12044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "ae579a30c3be4a848169035c059b5cca"
format_stamp: "Formatted at 2026-06-20 07:09:58 on dist-test-slave-4n6h"
server_key: "fe9cc7a86d803b589c3f68199f5f41f9"
server_key_iv: "85c55fcbb56fa828ae36341758dad769"
server_key_version: "encryptionkey@0"
I20260620 07:09:58.692795 12044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:58.729822 12044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:58.733793 12044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:58.734114 12044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:58.735117 12044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:58.736397 12044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:58.736475 12044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:58.736620 12044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:58.736689 12044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:58.768929 12044 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:36887
I20260620 07:09:58.769045 12171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:36887 every 8 connection(s)
I20260620 07:09:58.770206 12044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:09:58.772873 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 12044
I20260620 07:09:58.773118 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:09:58.773736 6357 external_mini_cluster.cc:1428] Setting key d4b6ed8247aa1172b6154233b5756bd3
Jun 20 07:09:58 dist-test-slave-4n6h krb5kdc[11938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939398, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:58.790022 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:58.772778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:60209 (local address 127.6.53.126:42149)
0620 07:09:58.773275 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:58.773282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:58.774323 (+ 1041us) server_negotiation.cc:407] Connection header received
0620 07:09:58.776309 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:58.776314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:58.776412 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:58.776528 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:58.778051 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:58.779197 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:58.780710 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:58.781113 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:58.781267 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:58.784604 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:58.784609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:58.784612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:58.784642 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:58.787109 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:58.787815 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:58.787822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:58.787830 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:58.787943 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:58.788365 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:58.788372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:58.788375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:58.788639 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:58.788802 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:58.789450 (+ 648us) server_negotiation.cc:299] Negotiation successful
0620 07:09:58.789674 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":234,"threads_started":1}
I20260620 07:09:58.790963 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:58.773326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42149 (local address 127.6.53.65:60209)
0620 07:09:58.774111 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:58.774155 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:58.776050 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:58.776773 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:58.776782 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:58.777265 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:58.777817 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:58.777834 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:58.779346 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:58.779352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:58.780565 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:58.780578 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:58.781334 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:58.781343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:58.781586 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:58.782322 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:58.782344 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:58.784398 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:58.787243 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:58.787252 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:58.787267 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:58.787680 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:58.788055 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:58.788061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:58.788065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:58.788253 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:58.788763 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:58.788770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:58.789097 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 07:09:58.789353 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 07:09:58.789847 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":357,"threads_started":1}
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
I20260620 07:09:58.792549 12172 heartbeater.cc:347] Connected to a master server at 127.6.53.126:42149
I20260620 07:09:58.792856 12172 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:58.793653 12172 heartbeater.cc:511] Master 127.6.53.126:42149 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
I20260620 07:09:58.795807 11983 ts_manager.cc:194] Registered new tserver with Master: ae579a30c3be4a848169035c059b5cca (127.6.53.65:36887)
I20260620 07:09:58.797364 11983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:60209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:09:58.816993 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42149
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:59.128341 12178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:59.128615 12178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:59.128804 12178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:59.135928 12178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:59.136029 12178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:59.136169 12178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:09:59.145545 12178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:59.146806 12178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:59.148106 12178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:59.161916 12184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:59.162575 12178 server_base.cc:1034] running on GCE node
W20260620 07:09:59.162631 12183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:59.166514 12186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:59.167327 12178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:59.168429 12178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:59.169649 12178 hybrid_clock.cc:648] HybridClock initialized: now 1781939399169604 us; error 33 us; skew 500 ppm
I20260620 07:09:59.170010 12178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:59 dist-test-slave-4n6h krb5kdc[11938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939399, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:59.173679 12178 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:09:59.177855 12178 webserver.cc:460] Webserver started at http://127.6.53.66:39365/ using document root <none> and password file <none>
I20260620 07:09:59.178560 12178 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:59.178663 12178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:59.178908 12178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:59.181660 12178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "50582c728833436c9cd37650ace966aa"
format_stamp: "Formatted at 2026-06-20 07:09:59 on dist-test-slave-4n6h"
server_key: "6c62e44447445515e4a5065921c93902"
server_key_iv: "1a321f2166ed29bd7570ba657ce4bec1"
server_key_version: "encryptionkey@0"
I20260620 07:09:59.182420 12178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "50582c728833436c9cd37650ace966aa"
format_stamp: "Formatted at 2026-06-20 07:09:59 on dist-test-slave-4n6h"
server_key: "6c62e44447445515e4a5065921c93902"
server_key_iv: "1a321f2166ed29bd7570ba657ce4bec1"
server_key_version: "encryptionkey@0"
I20260620 07:09:59.187358 12178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:09:59.190653 12193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:59.191924 12178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:09:59.192080 12178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "50582c728833436c9cd37650ace966aa"
format_stamp: "Formatted at 2026-06-20 07:09:59 on dist-test-slave-4n6h"
server_key: "6c62e44447445515e4a5065921c93902"
server_key_iv: "1a321f2166ed29bd7570ba657ce4bec1"
server_key_version: "encryptionkey@0"
I20260620 07:09:59.192276 12178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:59.225099 12178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:59.229564 12178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:59.229848 12178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:59.230903 12178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:59.232241 12178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:59.232313 12178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:59.232463 12178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:59.232522 12178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:59.264807 12178 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:33475
I20260620 07:09:59.264851 12305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:33475 every 8 connection(s)
I20260620 07:09:59.266119 12178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:09:59.275656 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 12178
I20260620 07:09:59.275822 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:09:59.276399 6357 external_mini_cluster.cc:1428] Setting key 4648ce6e6d6e7f3fce8f2c730be31328
Jun 20 07:09:59 dist-test-slave-4n6h krb5kdc[11938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939399, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:59.287184 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:59.268710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:45341 (local address 127.6.53.126:42149)
0620 07:09:59.268878 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:59.268883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:59.270028 (+ 1145us) server_negotiation.cc:407] Connection header received
0620 07:09:59.271860 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:59.271865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:59.271934 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:59.272030 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:59.273409 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.274091 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.275640 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.276307 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.276449 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:59.280297 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:59.280302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:59.280305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:59.280337 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:59.283704 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:59.284643 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:59.284649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:59.284654 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:59.284814 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:59.285232 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:59.285237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:59.285242 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:59.285624 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:59.285777 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:59.286482 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 07:09:59.286743 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:09:59.287823 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:59.269214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42149 (local address 127.6.53.66:45341)
0620 07:09:59.269871 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:59.269913 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:59.271541 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:59.272243 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:59.272250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:59.272730 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:59.273266 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:59.273283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.274207 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.274210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:59.275493 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:59.275504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.276539 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.276548 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:59.276763 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:59.277412 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:59.277438 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:59.280085 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:59.283832 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:59.283844 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:59.283863 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:59.284513 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:59.284919 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:59.284922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:59.284925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:59.285117 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:59.285741 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:59.285747 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:59.286073 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 07:09:59.286367 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 07:09:59.286739 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":278,"threads_started":1}
I20260620 07:09:59.289326 12306 heartbeater.cc:347] Connected to a master server at 127.6.53.126:42149
I20260620 07:09:59.289628 12306 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:59.290386 12306 heartbeater.cc:511] Master 127.6.53.126:42149 requested a full tablet report, sending...
I20260620 07:09:59.291850 11983 ts_manager.cc:194] Registered new tserver with Master: 50582c728833436c9cd37650ace966aa (127.6.53.66:33475)
I20260620 07:09:59.292656 11983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:45341
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:09:59.320637 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42149
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:09:59.637879 12311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:09:59.638196 12311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:09:59.638419 12311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:09:59.645541 12311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:09:59.645642 12311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:09:59.645839 12311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:09:59.655433 12311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:09:59.656689 12311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:09:59.658007 12311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:09:59.671080 12319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:59.671007 12317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:09:59.671111 12316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:09:59.671897 12311 server_base.cc:1034] running on GCE node
I20260620 07:09:59.672526 12311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:09:59.673485 12311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:09:59.674710 12311 hybrid_clock.cc:648] HybridClock initialized: now 1781939399674668 us; error 58 us; skew 500 ppm
I20260620 07:09:59.675032 12311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:09:59 dist-test-slave-4n6h krb5kdc[11938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939399, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:59.678143 12311 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:09:59.679631 12311 webserver.cc:460] Webserver started at http://127.6.53.67:40819/ using document root <none> and password file <none>
I20260620 07:09:59.680302 12311 fs_manager.cc:359] Metadata directory not provided
I20260620 07:09:59.680449 12311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:09:59.680751 12311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:09:59.683642 12311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "6aa15d821b8e436db2307f38841d7c88"
format_stamp: "Formatted at 2026-06-20 07:09:59 on dist-test-slave-4n6h"
server_key: "3e22a483dbc884182b03b0e5892f66da"
server_key_iv: "a674239ff03d63e801662ac62dbe3735"
server_key_version: "encryptionkey@0"
I20260620 07:09:59.684407 12311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "6aa15d821b8e436db2307f38841d7c88"
format_stamp: "Formatted at 2026-06-20 07:09:59 on dist-test-slave-4n6h"
server_key: "3e22a483dbc884182b03b0e5892f66da"
server_key_iv: "a674239ff03d63e801662ac62dbe3735"
server_key_version: "encryptionkey@0"
I20260620 07:09:59.689328 12311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 07:09:59.692832 12326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:59.694375 12311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:09:59.694543 12311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "6aa15d821b8e436db2307f38841d7c88"
format_stamp: "Formatted at 2026-06-20 07:09:59 on dist-test-slave-4n6h"
server_key: "3e22a483dbc884182b03b0e5892f66da"
server_key_iv: "a674239ff03d63e801662ac62dbe3735"
server_key_version: "encryptionkey@0"
I20260620 07:09:59.694746 12311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:09:59.709863 12311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:09:59.713953 12311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:09:59.714243 12311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:09:59.715240 12311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:09:59.716514 12311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:09:59.716622 12311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:59.716773 12311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:09:59.716847 12311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:09:59.749708 12311 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:36445
I20260620 07:09:59.749753 12438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:36445 every 8 connection(s)
I20260620 07:09:59.750995 12311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:09:59.759243 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 12311
I20260620 07:09:59.759430 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:09:59.759976 6357 external_mini_cluster.cc:1428] Setting key 14088ea9f1e2ae3201299acfa3054cf0
Jun 20 07:09:59 dist-test-slave-4n6h krb5kdc[11938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939399, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:09:59.771476 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:59.753625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:35371 (local address 127.6.53.126:42149)
0620 07:09:59.753814 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:59.753819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:59.754987 (+ 1168us) server_negotiation.cc:407] Connection header received
0620 07:09:59.756721 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:59.756725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:59.756827 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:59.756980 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:59.758450 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.759558 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.761290 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.761794 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.761917 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:59.765589 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:59.765594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:59.765598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:59.765629 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:59.768501 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:59.769228 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:59.769235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:59.769243 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:59.769363 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:59.769796 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:59.769805 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:59.769808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:59.770072 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:59.770224 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:59.770913 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 07:09:59.771135 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:09:59.772374 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:59.754071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42149 (local address 127.6.53.67:35371)
0620 07:09:59.754830 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 07:09:59.754866 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:09:59.756431 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:09:59.757170 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:09:59.757178 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:09:59.757682 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:09:59.758225 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:59.758241 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.759699 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.759702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:59.761126 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:09:59.761142 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.762018 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.762025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:09:59.762351 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:59.763100 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:09:59.763131 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:09:59.765380 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:09:59.768664 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:59.768673 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:59.768689 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:59.769093 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:59.769478 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:09:59.769483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:09:59.769487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:09:59.769676 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:09:59.770194 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:09:59.770202 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:09:59.770551 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 07:09:59.770803 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 07:09:59.771298 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":341,"threads_started":1}
I20260620 07:09:59.773947 12439 heartbeater.cc:347] Connected to a master server at 127.6.53.126:42149
I20260620 07:09:59.774289 12439 heartbeater.cc:464] Registering TS with master...
I20260620 07:09:59.775238 12439 heartbeater.cc:511] Master 127.6.53.126:42149 requested a full tablet report, sending...
I20260620 07:09:59.776697 11983 ts_manager.cc:194] Registered new tserver with Master: 6aa15d821b8e436db2307f38841d7c88 (127.6.53.67:36445)
I20260620 07:09:59.777469 11983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:35371
I20260620 07:09:59.787712 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:09:59.800618 12172 heartbeater.cc:503] Master 127.6.53.126:42149 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/test-admin.keytab.
Jun 20 07:09:59 dist-test-slave-4n6h krb5kdc[11938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:59.814957 6357 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 07:09:59.819909 12446 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:59.817526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.6.53.65:36887)
0620 07:09:59.818195 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:59.818202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:59.818226 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:09:59.818347 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:59.818354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:59.818562 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:59.818703 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:59.819512 (+ 809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53868: BlockingRecv error: recv got EOF from 127.0.0.1:53868 (error 108)
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":368,"threads_started":1}
W20260620 07:09:59.819924 6357 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/krb5cc' of type FILE
Jun 20 07:09:59 dist-test-slave-4n6h krb5kdc[11938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:59.821720 6357 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 07:09:59 dist-test-slave-4n6h krb5kdc[11938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:59.823086 6357 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 07:09:59 dist-test-slave-4n6h krb5kdc[11938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:09:59.839536 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:59.824687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53882 (local address 127.6.53.65:36887)
0620 07:09:59.824857 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:59.824864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:59.825264 (+ 400us) server_negotiation.cc:407] Connection header received
0620 07:09:59.825444 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:59.825448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:59.825503 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:59.825602 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:59.826623 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.827528 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.828828 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.829399 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.829527 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:59.833061 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:59.833071 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:59.833085 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:59.833122 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:59.836080 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:59.836934 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:59.836960 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:59.836965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:59.837140 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:59.837606 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:59.837612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:59.837616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:59.838120 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:59.838501 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:59.838864 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 07:09:59.839128 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 07:09:59.845556 12446 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:59.843761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.6.53.65:36887)
0620 07:09:59.843979 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:59.843986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:59.844253 (+ 267us) server_negotiation.cc:407] Connection header received
0620 07:09:59.844437 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:59.844442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:59.844511 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:59.844615 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:59.845290 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53896: BlockingRecv error: recv got EOF from 127.0.0.1:53896 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 07:09:59.845992 6357 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/krb5cc)
Jun 20 07:09:59 dist-test-slave-4n6h krb5kdc[11938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:59.847487 6357 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 07:09:59 dist-test-slave-4n6h krb5kdc[11938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:59.848744 6357 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 07:09:59 dist-test-slave-4n6h krb5kdc[11938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:09:59.861987 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:59.850074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53910 (local address 127.6.53.65:36887)
0620 07:09:59.850300 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:59.850306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:59.850431 (+ 125us) server_negotiation.cc:407] Connection header received
0620 07:09:59.850549 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:59.850552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:59.850600 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:59.850673 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:59.851674 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.852331 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.853473 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.853896 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.853984 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:59.857413 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:59.857417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:59.857421 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:59.857457 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:59.859768 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:59.860341 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:59.860345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:59.860347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:59.860458 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:59.860766 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:59.860769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:59.860771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:59.861000 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:59.861203 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:59.861502 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 07:09:59.861663 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 07:09:59.866235 12446 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:09:59.864485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.6.53.65:36887)
0620 07:09:59.864665 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:59.864673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:59.865083 (+ 410us) server_negotiation.cc:407] Connection header received
0620 07:09:59.865182 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:59.865193 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:59.865270 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:59.865369 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:59.866004 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53924: BlockingRecv error: recv got EOF from 127.0.0.1:53924 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 07:09:59.866382 6357 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939352012798-6357-0/krb5kdc/krb5cc' of type FILE
Jun 20 07:09:59 dist-test-slave-4n6h krb5kdc[11938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:59.867794 6357 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 07:09:59 dist-test-slave-4n6h krb5kdc[11938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:09:59.868979 6357 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 07:09:59 dist-test-slave-4n6h krb5kdc[11938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:09:59.882823 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:59.870351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.6.53.65:36887)
0620 07:09:59.870543 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:59.870551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:59.870788 (+ 237us) server_negotiation.cc:407] Connection header received
0620 07:09:59.870959 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:59.870964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:59.871026 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:59.871120 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:59.872053 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.872747 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.873920 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.874423 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.874554 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:59.878066 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:59.878072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:59.878077 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:59.878112 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:59.880653 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:59.881201 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:59.881204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:59.881207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:59.881347 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:59.881674 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:59.881677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:59.881679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:59.881925 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:59.882166 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:59.882424 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 07:09:59.882574 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:09:59.884470 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 12044
I20260620 07:09:59.896184 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:59.885120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.6.53.65:36887)
0620 07:09:59.885304 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:59.885311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:59.885606 (+ 295us) server_negotiation.cc:407] Connection header received
0620 07:09:59.885702 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:59.885705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:59.885754 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:59.885840 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:59.886814 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.887451 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.888543 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.888932 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.889022 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:09:59.891098 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:09:59.891102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:09:59.891105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:09:59.891132 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:09:59.893542 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:59.894241 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:59.894244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:59.894246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:59.894415 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:09:59.894744 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:09:59.894747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:09:59.894749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:09:59.894998 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:09:59.895266 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 07:09:59.895585 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 07:09:59.895799 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:09:59.896835 12167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:09:59.978433 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 12044
I20260620 07:09:59.994132 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 12178
Jun 20 07:09:59 dist-test-slave-4n6h krb5kdc[11938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:10:00.010241 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:09:59.995073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.6.53.66:33475)
0620 07:09:59.995562 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 07:09:59.995569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:09:59.995595 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:09:59.995651 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:09:59.995657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:09:59.995916 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:09:59.996044 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:09:59.997026 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.998120 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:09:59.999473 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:09:59.999994 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:00.000088 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:00.004096 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:00.004111 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:00.004128 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:00.004171 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:00.007427 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:00.008199 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:00.008207 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:00.008209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:00.008348 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:00.008722 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:00.008725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:00.008727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:00.009125 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:00.009371 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:00.009704 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 07:10:00.009915 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":241,"threads_started":1}
I20260620 07:10:00.011819 12301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:00.094466 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 12178
I20260620 07:10:00.110318 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 12311
Jun 20 07:10:00 dist-test-slave-4n6h krb5kdc[11938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:10:00.127777 12461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:00.111472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.6.53.67:36445)
0620 07:10:00.112236 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:00.112243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:00.112273 (+ 30us) server_negotiation.cc:407] Connection header received
0620 07:10:00.112332 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:00.112338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:00.112558 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:00.112733 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:00.114128 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:00.115258 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:00.116628 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:00.117234 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:00.117407 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:00.121689 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:00.121702 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:00.121718 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:00.121752 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:00.124747 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:00.125626 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:00.125634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:00.125636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:00.125760 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:00.126153 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:00.126160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:00.126163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:00.126565 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:00.126921 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:00.127199 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 07:10:00.127465 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":432,"threads_started":1}
I20260620 07:10:00.129310 12434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:00.213832 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 12311
I20260620 07:10:00.229626 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 11951
I20260620 07:10:00.230892 12013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:00.331422 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 11951
2026-06-20T07:10:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3865 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:10:00 dist-test-slave-4n6h krb5kdc[12466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:10:00 dist-test-slave-4n6h krb5kdc[12466](info): set up 2 sockets
Jun 20 07:10:00 dist-test-slave-4n6h krb5kdc[12466](info): commencing operation
krb5kdc: starting...
W20260620 07:10:01.526891 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:10:01 dist-test-slave-4n6h krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:10:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:10:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.126.keytab.
Entry for principal oryx/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.126.keytab.
I20260620 07:10:01.733754 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:41085
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.126.keytab
--principal=oryx/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:41085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:02.055459 12479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:02.055730 12479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:02.055897 12479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:02.063009 12479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:10:02.063113 12479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:02.063218 12479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:02.063313 12479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:10:02.063401 12479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:10:02.073205 12479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:41085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.126.keytab
--principal=oryx/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:41085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:02.074548 12479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:02.075848 12479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:02.087755 12485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:02.088012 12487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:02.087754 12484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:02.089365 12479 server_base.cc:1034] running on GCE node
I20260620 07:10:02.090173 12479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:02.091645 12479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:02.092851 12479 hybrid_clock.cc:648] HybridClock initialized: now 1781939402092803 us; error 38 us; skew 500 ppm
I20260620 07:10:02.093223 12479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:02 dist-test-slave-4n6h krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939402, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:02.096362 12479 init.cc:377] Logged in from keytab as oryx/127.6.53.126@KRBTEST.COM (short username oryx)
I20260620 07:10:02.097862 12479 webserver.cc:460] Webserver started at http://127.6.53.126:33663/ using document root <none> and password file <none>
I20260620 07:10:02.098624 12479 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:02.098770 12479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:02.099073 12479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:02.101962 12479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "8f0bb53d5db24273b564b44d83a922bc"
format_stamp: "Formatted at 2026-06-20 07:10:02 on dist-test-slave-4n6h"
server_key: "70f74954fbc49acc2cb1d53048b5cf01"
server_key_iv: "2d65fc1388b7fe3c6b56f87cf1c8966b"
server_key_version: "encryptionkey@0"
I20260620 07:10:02.102840 12479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "8f0bb53d5db24273b564b44d83a922bc"
format_stamp: "Formatted at 2026-06-20 07:10:02 on dist-test-slave-4n6h"
server_key: "70f74954fbc49acc2cb1d53048b5cf01"
server_key_iv: "2d65fc1388b7fe3c6b56f87cf1c8966b"
server_key_version: "encryptionkey@0"
I20260620 07:10:02.107901 12479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:10:02.111258 12494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:02.112560 12479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:10:02.112737 12479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "8f0bb53d5db24273b564b44d83a922bc"
format_stamp: "Formatted at 2026-06-20 07:10:02 on dist-test-slave-4n6h"
server_key: "70f74954fbc49acc2cb1d53048b5cf01"
server_key_iv: "2d65fc1388b7fe3c6b56f87cf1c8966b"
server_key_version: "encryptionkey@0"
I20260620 07:10:02.112927 12479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:02.130060 12479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:02.134721 12479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:02.134995 12479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:02.153043 12479 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:41085
I20260620 07:10:02.153076 12545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:41085 every 8 connection(s)
I20260620 07:10:02.154480 12479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:10:02.158330 12546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:02.163522 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 12479
I20260620 07:10:02.163733 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:10:02.164391 6357 external_mini_cluster.cc:1428] Setting key 5add637ed1eeb0e6069bff1a629fe52b
I20260620 07:10:02.170816 12546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc: Bootstrap starting.
I20260620 07:10:02.174240 12546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:02.175377 12546 log.cc:824] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:02.178296 12546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc: No bootstrap required, opened a new log
Jun 20 07:10:02 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM
I20260620 07:10:02.187115 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:02.167064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.6.53.126:41085)
0620 07:10:02.168101 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:02.168122 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:02.168175 (+ 53us) server_negotiation.cc:407] Connection header received
0620 07:10:02.170062 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:02.170086 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:02.170542 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:02.170916 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:02.172331 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:02.173636 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:02.175027 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:02.175742 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:02.175941 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:02.179852 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:02.179860 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:02.179876 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:02.179909 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:02.183132 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:02.183793 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:02.183799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:02.183804 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:02.183960 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:02.184348 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:02.184352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:02.184354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:02.184852 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:02.185231 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:02.185640 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 07:10:02.186055 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":446,"threads_started":1}
I20260620 07:10:02.186986 12546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } }
I20260620 07:10:02.187472 12546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:02.187610 12546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0bb53d5db24273b564b44d83a922bc, State: Initialized, Role: FOLLOWER
I20260620 07:10:02.188206 12546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } }
I20260620 07:10:02.188472 12546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:10:02.188632 12546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:10:02.188841 12546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:02.191450 12546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } }
I20260620 07:10:02.192014 12546 leader_election.cc:302] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f0bb53d5db24273b564b44d83a922bc; no voters:
I20260620 07:10:02.192615 12546 leader_election.cc:288] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:10:02.192718 12551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:10:02.193610 12551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [term 1 LEADER]: Becoming Leader. State: Replica: 8f0bb53d5db24273b564b44d83a922bc, State: Running, Role: LEADER
I20260620 07:10:02.194130 12551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } }
I20260620 07:10:02.194988 12546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:10:02.196863 12552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } }
I20260620 07:10:02.197460 12552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:02.196979 12553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f0bb53d5db24273b564b44d83a922bc. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } }
I20260620 07:10:02.197938 12553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:02.198887 12560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:10:02.206523 12560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:10:02.222536 12560 catalog_manager.cc:1347] Generated new cluster ID: 164f487c8df14824bf1945fe4a35bc1b
I20260620 07:10:02.222687 12560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:10:02.240650 12560 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:10:02.242501 12560 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:10:02.251490 12560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc: Generated new TSK 0
I20260620 07:10:02.252560 12560 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.65.keytab.
Entry for principal oryx/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.65.keytab.
I20260620 07:10:02.313352 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41085
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.65.keytab
--principal=oryx/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:02.639906 12572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:02.640177 12572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:02.640354 12572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:02.647516 12572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:02.647667 12572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:02.647877 12572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:10:02.657605 12572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.65.keytab
--principal=oryx/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:02.658952 12572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:02.660279 12572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:02.673449 12577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:02.673466 12578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:02.673512 12580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:02.674024 12572 server_base.cc:1034] running on GCE node
I20260620 07:10:02.674762 12572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:02.675658 12572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:02.676877 12572 hybrid_clock.cc:648] HybridClock initialized: now 1781939402676833 us; error 62 us; skew 500 ppm
I20260620 07:10:02.677184 12572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:02 dist-test-slave-4n6h krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939402, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:02.680262 12572 init.cc:377] Logged in from keytab as oryx/127.6.53.65@KRBTEST.COM (short username oryx)
I20260620 07:10:02.681569 12572 webserver.cc:460] Webserver started at http://127.6.53.65:43633/ using document root <none> and password file <none>
I20260620 07:10:02.682204 12572 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:02.682366 12572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:02.682624 12572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:02.685494 12572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "659e9bb195db4166b7c0c6bc30452fbc"
format_stamp: "Formatted at 2026-06-20 07:10:02 on dist-test-slave-4n6h"
server_key: "8a50ce9e1ce6bb9609a6ceb5bd7fa5b4"
server_key_iv: "fc29487bcd329035fdd27dfa6af70f5f"
server_key_version: "encryptionkey@0"
I20260620 07:10:02.686228 12572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "659e9bb195db4166b7c0c6bc30452fbc"
format_stamp: "Formatted at 2026-06-20 07:10:02 on dist-test-slave-4n6h"
server_key: "8a50ce9e1ce6bb9609a6ceb5bd7fa5b4"
server_key_iv: "fc29487bcd329035fdd27dfa6af70f5f"
server_key_version: "encryptionkey@0"
I20260620 07:10:02.691515 12572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 07:10:02.695170 12587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:02.696753 12572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 07:10:02.696925 12572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "659e9bb195db4166b7c0c6bc30452fbc"
format_stamp: "Formatted at 2026-06-20 07:10:02 on dist-test-slave-4n6h"
server_key: "8a50ce9e1ce6bb9609a6ceb5bd7fa5b4"
server_key_iv: "fc29487bcd329035fdd27dfa6af70f5f"
server_key_version: "encryptionkey@0"
I20260620 07:10:02.697118 12572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:02.710672 12572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:02.714478 12572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:02.714787 12572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:02.715730 12572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:02.717190 12572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:02.717262 12572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:02.717406 12572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:02.717507 12572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:02.753756 12572 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:42627
I20260620 07:10:02.753796 12699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:42627 every 8 connection(s)
I20260620 07:10:02.755281 12572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:10:02.755784 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 12572
I20260620 07:10:02.756016 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:10:02.756863 6357 external_mini_cluster.cc:1428] Setting key a07ae4b436cc91bc238ce49f97558f9e
Jun 20 07:10:02 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939402, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.6.53.66@KRBTEST.COM; defaulting to no policy
I20260620 07:10:02.776854 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:02.758297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:33615 (local address 127.6.53.126:41085)
0620 07:10:02.758779 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:02.758785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:02.760000 (+ 1215us) server_negotiation.cc:407] Connection header received
0620 07:10:02.761957 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:02.761963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:02.762065 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:02.762192 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:02.763824 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:02.764615 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:02.766320 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:02.766782 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:02.766937 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:02.770640 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:02.770645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:02.770649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:02.770682 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:02.773621 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:02.774499 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:02.774504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:02.774506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:02.774634 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:02.775144 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:02.775152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:02.775159 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:02.775469 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:02.775598 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:02.776389 (+ 791us) server_negotiation.cc:299] Negotiation successful
0620 07:10:02.776575 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":249,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.53.66@KRBTEST.COM" created.
I20260620 07:10:02.778090 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:02.758973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.65:33615)
0620 07:10:02.759826 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:02.759871 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:02.761666 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:02.762486 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:02.762495 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:02.763041 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:02.763641 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:02.763659 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:02.764795 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:02.764801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:02.766133 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:02.766146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:02.767045 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:02.767053 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:02.767300 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:02.768042 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:02.768066 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:02.770410 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:02.773789 (+ 3379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:02.773800 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:02.773818 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:02.774342 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:02.774757 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:02.774763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:02.774768 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:02.774973 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:02.775596 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:02.775602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:02.776005 (+ 403us) client_negotiation.cc:769] Sending connection context
0620 07:10:02.776288 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 07:10:02.776697 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":384,"threads_started":1}
I20260620 07:10:02.779716 12700 heartbeater.cc:347] Connected to a master server at 127.6.53.126:41085
I20260620 07:10:02.780037 12700 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:02.780910 12700 heartbeater.cc:511] Master 127.6.53.126:41085 requested a full tablet report, sending...
I20260620 07:10:02.783592 12511 ts_manager.cc:194] Registered new tserver with Master: 659e9bb195db4166b7c0c6bc30452fbc (127.6.53.65:42627)
I20260620 07:10:02.786114 12511 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:33615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.66.keytab.
Entry for principal oryx/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.66.keytab.
I20260620 07:10:02.803799 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41085
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.66.keytab
--principal=oryx/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:03.134690 12706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:03.134966 12706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:03.135135 12706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:03.142618 12706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:03.142735 12706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:03.142925 12706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:10:03.152931 12706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.66.keytab
--principal=oryx/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:03.154222 12706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:03.155617 12706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:03.168980 12712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:03.169335 12711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:03.169276 12714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:03.170557 12706 server_base.cc:1034] running on GCE node
I20260620 07:10:03.171437 12706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:03.172751 12706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:03.173997 12706 hybrid_clock.cc:648] HybridClock initialized: now 1781939403173953 us; error 43 us; skew 500 ppm
I20260620 07:10:03.174431 12706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:03 dist-test-slave-4n6h krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939403, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:03.177868 12706 init.cc:377] Logged in from keytab as oryx/127.6.53.66@KRBTEST.COM (short username oryx)
I20260620 07:10:03.179368 12706 webserver.cc:460] Webserver started at http://127.6.53.66:42005/ using document root <none> and password file <none>
I20260620 07:10:03.180052 12706 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:03.180151 12706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:03.180395 12706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:03.183410 12706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "42141b2ca9ed4593a530955665d7a0fc"
format_stamp: "Formatted at 2026-06-20 07:10:03 on dist-test-slave-4n6h"
server_key: "e703b6298ccb4e2ed5fe96a9c45bb754"
server_key_iv: "b98791a764e9049c95bb4b1752a05b06"
server_key_version: "encryptionkey@0"
I20260620 07:10:03.184120 12706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "42141b2ca9ed4593a530955665d7a0fc"
format_stamp: "Formatted at 2026-06-20 07:10:03 on dist-test-slave-4n6h"
server_key: "e703b6298ccb4e2ed5fe96a9c45bb754"
server_key_iv: "b98791a764e9049c95bb4b1752a05b06"
server_key_version: "encryptionkey@0"
I20260620 07:10:03.189309 12706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 07:10:03.192827 12721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:03.194187 12706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:10:03.194430 12706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "42141b2ca9ed4593a530955665d7a0fc"
format_stamp: "Formatted at 2026-06-20 07:10:03 on dist-test-slave-4n6h"
server_key: "e703b6298ccb4e2ed5fe96a9c45bb754"
server_key_iv: "b98791a764e9049c95bb4b1752a05b06"
server_key_version: "encryptionkey@0"
I20260620 07:10:03.194605 12706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:03.220906 12706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:03.225682 12706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:03.225991 12706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:03.227102 12706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:03.228475 12706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:03.228555 12706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:03.228699 12706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:03.228749 12706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:03.263087 12706 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:39055
I20260620 07:10:03.263144 12833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:39055 every 8 connection(s)
I20260620 07:10:03.264331 12706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:10:03.274585 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 12706
I20260620 07:10:03.274783 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:10:03.275236 6357 external_mini_cluster.cc:1428] Setting key cd299c03a6e16404ffd4bc83ee719d7e
Jun 20 07:10:03 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939403, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM
I20260620 07:10:03.284461 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:03.266925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:32791 (local address 127.6.53.126:41085)
0620 07:10:03.267102 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:03.267107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:03.268349 (+ 1242us) server_negotiation.cc:407] Connection header received
0620 07:10:03.270106 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:03.270111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:03.270203 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:03.270366 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:03.271902 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:03.272607 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:03.274180 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:03.274663 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:03.274799 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:03.278545 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:03.278550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:03.278558 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:03.278590 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:03.281133 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:03.282033 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:03.282037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:03.282040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:03.282160 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:03.282644 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:03.282651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:03.282659 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:03.282995 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:03.283151 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:03.283854 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 07:10:03.284112 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:10:03.285534 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:03.267424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.66:32791)
0620 07:10:03.268171 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:03.268213 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:03.269841 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:03.270584 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:03.270592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:03.271093 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:03.271681 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:03.271735 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:03.272761 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:03.272766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:03.274034 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:03.274046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:03.274893 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:03.274900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:03.275128 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:03.275784 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:03.275886 (+ 102us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:03.278323 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:03.281283 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:03.281293 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:03.281311 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:03.281833 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:03.282299 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:03.282306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:03.282311 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:03.282518 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:03.283125 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:03.283133 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:03.283486 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 07:10:03.283781 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 07:10:03.284327 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":308,"threads_started":1}
I20260620 07:10:03.287191 12834 heartbeater.cc:347] Connected to a master server at 127.6.53.126:41085
I20260620 07:10:03.287513 12834 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:03.288228 12834 heartbeater.cc:511] Master 127.6.53.126:41085 requested a full tablet report, sending...
I20260620 07:10:03.289745 12511 ts_manager.cc:194] Registered new tserver with Master: 42141b2ca9ed4593a530955665d7a0fc (127.6.53.66:39055)
I20260620 07:10:03.290999 12511 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:32791
WARNING: no policy specified for oryx/127.6.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.67.keytab.
Entry for principal oryx/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.67.keytab.
I20260620 07:10:03.321424 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41085
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.67.keytab
--principal=oryx/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:03.646693 12839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:03.646979 12839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:03.647132 12839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:03.654546 12839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:03.654654 12839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:03.654832 12839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:10:03.664633 12839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.67.keytab
--principal=oryx/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:03.665977 12839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:03.667451 12839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:03.683607 12844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:03.683563 12847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:03.683647 12845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:03.684643 12839 server_base.cc:1034] running on GCE node
I20260620 07:10:03.685606 12839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:03.686925 12839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:03.688194 12839 hybrid_clock.cc:648] HybridClock initialized: now 1781939403688121 us; error 36 us; skew 500 ppm
I20260620 07:10:03.688647 12839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:03 dist-test-slave-4n6h krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939403, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:03.696095 12839 init.cc:377] Logged in from keytab as oryx/127.6.53.67@KRBTEST.COM (short username oryx)
I20260620 07:10:03.697844 12839 webserver.cc:460] Webserver started at http://127.6.53.67:45071/ using document root <none> and password file <none>
I20260620 07:10:03.698885 12839 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:03.699038 12839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:03.699450 12839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:03.703353 12839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "ca2ec70afce24c1099d2f07d015010ec"
format_stamp: "Formatted at 2026-06-20 07:10:03 on dist-test-slave-4n6h"
server_key: "7c75f98c9c53bb73b41e976990882d69"
server_key_iv: "ad62f4544c8de45bc96b3501ce3145fc"
server_key_version: "encryptionkey@0"
I20260620 07:10:03.704399 12839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "ca2ec70afce24c1099d2f07d015010ec"
format_stamp: "Formatted at 2026-06-20 07:10:03 on dist-test-slave-4n6h"
server_key: "7c75f98c9c53bb73b41e976990882d69"
server_key_iv: "ad62f4544c8de45bc96b3501ce3145fc"
server_key_version: "encryptionkey@0"
I20260620 07:10:03.712281 12839 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20260620 07:10:03.716496 12854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:03.718228 12839 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 07:10:03.718487 12839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "ca2ec70afce24c1099d2f07d015010ec"
format_stamp: "Formatted at 2026-06-20 07:10:03 on dist-test-slave-4n6h"
server_key: "7c75f98c9c53bb73b41e976990882d69"
server_key_iv: "ad62f4544c8de45bc96b3501ce3145fc"
server_key_version: "encryptionkey@0"
I20260620 07:10:03.718710 12839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:03.737563 12839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:03.742131 12839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:03.742449 12839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:03.743396 12839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:03.744704 12839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:03.744804 12839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:03.744961 12839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:03.745045 12839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:03.778801 12839 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:42697
I20260620 07:10:03.778844 12966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:42697 every 8 connection(s)
I20260620 07:10:03.780114 12839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:10:03.781896 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 12839
I20260620 07:10:03.782120 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:10:03.782663 6357 external_mini_cluster.cc:1428] Setting key 565fd3a6b67991599e34bd43baa20743
I20260620 07:10:03.789284 12700 heartbeater.cc:503] Master 127.6.53.126:41085 was elected leader, sending a full tablet report...
Jun 20 07:10:03 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939403, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM
I20260620 07:10:03.801097 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:03.782760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:44635 (local address 127.6.53.126:41085)
0620 07:10:03.782993 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:03.783000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:03.784376 (+ 1376us) server_negotiation.cc:407] Connection header received
0620 07:10:03.786487 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:03.786491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:03.786584 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:03.786670 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:03.788161 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:03.788902 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:03.791015 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:03.791454 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:03.791564 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:03.794926 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:03.794931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:03.794934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:03.794973 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:03.798052 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:03.798887 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:03.798891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:03.798893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:03.799078 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:03.799443 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:03.799446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:03.799448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:03.799786 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:03.799919 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:03.800627 (+ 708us) server_negotiation.cc:299] Negotiation successful
0620 07:10:03.800786 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 07:10:03.802124 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:03.783341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.67:44635)
0620 07:10:03.784206 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:03.784257 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:03.786166 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:03.786875 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:03.786883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:03.787411 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:03.788010 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:03.788029 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:03.789050 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:03.789054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:03.790840 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:03.790857 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:03.791651 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:03.791656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:03.791843 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:03.792461 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:03.792489 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:03.794737 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:03.798205 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:03.798215 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:03.798232 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:03.798777 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:03.799188 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:03.799192 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:03.799194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:03.799362 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:03.799911 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:03.799917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:03.800265 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 07:10:03.800541 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 07:10:03.800932 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":338,"threads_started":1}
I20260620 07:10:03.803519 12967 heartbeater.cc:347] Connected to a master server at 127.6.53.126:41085
I20260620 07:10:03.803819 12967 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:03.804540 12967 heartbeater.cc:511] Master 127.6.53.126:41085 requested a full tablet report, sending...
I20260620 07:10:03.805914 12511 ts_manager.cc:194] Registered new tserver with Master: ca2ec70afce24c1099d2f07d015010ec (127.6.53.67:42697)
I20260620 07:10:03.806871 12511 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:44635
I20260620 07:10:03.810531 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.53.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.53.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.53.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.125.keytab.
Entry for principal oryx/127.6.53.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.125.keytab.
I20260620 07:10:03.861447 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.125:38109
--webserver_interface=127.6.53.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.125.keytab
--principal=oryx/127.6.53.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:41085,127.6.53.125:38109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:04.181403 12972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:04.181685 12972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:04.181864 12972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:04.189049 12972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:10:04.189182 12972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:04.189288 12972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:04.189368 12972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:10:04.189430 12972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:10:04.199666 12972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:41085,127.6.53.125:38109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.125.keytab
--principal=oryx/127.6.53.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.125:38109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.53.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
W20260620 07:10:04.200052 12972 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.53.126:41085,127.6.53.125:38109'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 07:10:04.201117 12972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:04.202620 12972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:04.214421 12977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:04.214382 12980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:04.214481 12978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:04.221200 12972 server_base.cc:1034] running on GCE node
I20260620 07:10:04.222029 12972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:04.223721 12972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:04.224932 12972 hybrid_clock.cc:648] HybridClock initialized: now 1781939404224848 us; error 62 us; skew 500 ppm
I20260620 07:10:04.225261 12972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:04 dist-test-slave-4n6h krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939404, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:04.228410 12972 init.cc:377] Logged in from keytab as oryx/127.6.53.125@KRBTEST.COM (short username oryx)
I20260620 07:10:04.229898 12972 webserver.cc:460] Webserver started at http://127.6.53.125:37591/ using document root <none> and password file <none>
I20260620 07:10:04.230683 12972 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:04.230798 12972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:04.231149 12972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:04.234025 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/data/instance:
uuid: "0963332cf16b489e9eaf2607296f0e60"
format_stamp: "Formatted at 2026-06-20 07:10:04 on dist-test-slave-4n6h"
server_key: "9758648b5a53df4b71becef9ebe2b028"
server_key_iv: "633654aa52feb5a8894dafea70ca9046"
server_key_version: "encryptionkey@0"
I20260620 07:10:04.234827 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/wal/instance:
uuid: "0963332cf16b489e9eaf2607296f0e60"
format_stamp: "Formatted at 2026-06-20 07:10:04 on dist-test-slave-4n6h"
server_key: "9758648b5a53df4b71becef9ebe2b028"
server_key_iv: "633654aa52feb5a8894dafea70ca9046"
server_key_version: "encryptionkey@0"
I20260620 07:10:04.239959 12972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:10:04.243637 12987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:04.245030 12972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:10:04.245216 12972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/wal
uuid: "0963332cf16b489e9eaf2607296f0e60"
format_stamp: "Formatted at 2026-06-20 07:10:04 on dist-test-slave-4n6h"
server_key: "9758648b5a53df4b71becef9ebe2b028"
server_key_iv: "633654aa52feb5a8894dafea70ca9046"
server_key_version: "encryptionkey@0"
I20260620 07:10:04.245435 12972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:04.257973 12972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:04.261900 12972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:04.262180 12972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:04.281577 12972 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.125:38109
I20260620 07:10:04.281669 13038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.125:38109 every 8 connection(s)
I20260620 07:10:04.283243 12972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/data/info.pb
I20260620 07:10:04.287129 13039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:04.291150 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 12972
I20260620 07:10:04.293495 12834 heartbeater.cc:503] Master 127.6.53.126:41085 was elected leader, sending a full tablet report...
I20260620 07:10:04.295221 13039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:10:04 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.53.125@KRBTEST.COM
Jun 20 07:10:04 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939404, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.125@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM
I20260620 07:10:04.315871 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.299034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39522 (local address 127.6.53.126:41085)
0620 07:10:04.299212 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:04.299216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:04.300194 (+ 978us) server_negotiation.cc:407] Connection header received
0620 07:10:04.300450 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:04.300456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:04.300533 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:04.300638 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:04.301898 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.302957 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.304895 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.305387 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.305481 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.309574 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:04.309579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:04.309582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:04.309614 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:04.312795 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.313599 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.313603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.313605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.313742 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.314115 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.314118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.314120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.314572 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:04.314717 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:04.315336 (+ 619us) server_negotiation.cc:299] Negotiation successful
0620 07:10:04.315540 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:10:04.316895 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.293516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56360 (local address 127.6.53.125:38109)
0620 07:10:04.294421 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:04.294433 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:04.294475 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:10:04.296384 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:04.296408 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:04.296840 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:04.297260 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:04.299224 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.301155 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.302791 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.303818 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.304005 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.309043 (+ 5038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:04.309053 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:04.309067 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:04.309109 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:04.311982 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.312787 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.312793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.312799 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.313014 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.313395 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.313406 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.313410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.314161 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:04.314434 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:04.314974 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 07:10:04.315433 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":627,"spinlock_wait_cycles":20864,"thread_start_us":393,"threads_started":1}
I20260620 07:10:04.316931 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.299130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.0.0.1:39522)
0620 07:10:04.300037 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:04.300073 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:04.300310 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:04.300719 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:04.300727 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:04.301483 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:04.301718 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.301727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.303112 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.303117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.304750 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.304766 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.305631 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.305640 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.305843 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.306541 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:04.306575 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:04.309392 (+ 2817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:04.312902 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:04.312912 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:04.312919 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:04.313468 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:04.313862 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:04.313866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:04.313870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:04.314010 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:04.314712 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:04.314721 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:04.314925 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 07:10:04.315132 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 07:10:04.316316 (+ 1184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":864,"thread_start_us":482,"threads_started":1}
I20260620 07:10:04.318819 13039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:10:04 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939404, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.125@KRBTEST.COM for oryx/127.6.53.125@KRBTEST.COM
I20260620 07:10:04.331092 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.319489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:38109 (local address 127.0.0.1:56372)
0620 07:10:04.319701 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:04.319722 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:04.319812 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:04.320227 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:04.320230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:04.320570 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:04.320688 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.320695 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.321646 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.321651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.322640 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.322652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.323340 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.323345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.323522 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.324081 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:04.324097 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:04.326117 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:04.328804 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:04.328812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:04.328815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:04.329159 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:04.329498 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:04.329506 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:04.329510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:04.329629 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:04.330111 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:04.330116 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:04.330298 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 07:10:04.330450 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 07:10:04.330744 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 07:10:04.331204 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.319573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.6.53.125:38109)
0620 07:10:04.320019 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:04.320023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:04.320035 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:10:04.320065 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:04.320068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:04.320115 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:04.320214 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:04.320818 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.321509 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.322770 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.323203 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.323340 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.326336 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:04.326341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:04.326345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:04.326371 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:04.328682 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.329268 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.329276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.329280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.329390 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.329727 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.329735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.329738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.329999 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:04.330147 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:04.330540 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 07:10:04.330869 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":349}
I20260620 07:10:04.335806 13039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: Bootstrap starting.
I20260620 07:10:04.338889 13039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:04.339897 13039 log.cc:824] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:04.342530 13039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: No bootstrap required, opened a new log
I20260620 07:10:04.349262 13039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } }
I20260620 07:10:04.349833 13039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:04.349920 13039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0963332cf16b489e9eaf2607296f0e60, State: Initialized, Role: FOLLOWER
I20260620 07:10:04.350535 13039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } }
I20260620 07:10:04.351312 13045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } } }
I20260620 07:10:04.351658 13045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:10:04.352313 13039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 07:10:04.361182 13056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:10:04.361336 13056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:10:04.362987 12972 master_runner.cc:428] Detected that this master 0963332cf16b489e9eaf2607296f0e60 is joining an existing cluster
I20260620 07:10:04.363099 12972 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.53.125:38109
I20260620 07:10:04.384479 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.367896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56376 (local address 127.6.53.125:38109)
0620 07:10:04.368047 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:04.368055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:04.368364 (+ 309us) server_negotiation.cc:407] Connection header received
0620 07:10:04.368558 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:04.368563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:04.368621 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:04.368733 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:04.369748 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.370955 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.373299 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.374026 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.374106 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.377301 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:04.377309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:04.377313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:04.377349 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:04.381413 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.382209 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.382214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.382217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.382379 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.382746 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.382751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.382753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.383101 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:04.383221 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:04.383844 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 07:10:04.384095 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:10:04.384478 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.367000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39524 (local address 127.6.53.126:41085)
0620 07:10:04.367229 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:04.367238 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:04.367729 (+ 491us) server_negotiation.cc:407] Connection header received
0620 07:10:04.368031 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:04.368037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:04.368120 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:04.368193 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:04.369886 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.370981 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.373313 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.373989 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.374103 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.377328 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:04.377334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:04.377338 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:04.377376 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:04.381211 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.382081 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.382086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.382090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.382287 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.382764 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.382769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.382772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.383116 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:04.383221 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:04.383912 (+ 691us) server_negotiation.cc:299] Negotiation successful
0620 07:10:04.384122 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:10:04.385571 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.366938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.0.0.1:39524)
0620 07:10:04.367600 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:04.367625 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:04.367866 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:04.368406 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:04.368412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:04.369168 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:04.369646 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.369659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.371140 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.371146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.373151 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.373167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.374455 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.374462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.374736 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.375662 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:04.375695 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:04.377118 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:04.381397 (+ 4279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:04.381401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:04.381405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:04.381937 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:04.382442 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:04.382453 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:04.382458 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:04.382626 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:04.383298 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:04.383302 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:04.383596 (+ 294us) client_negotiation.cc:769] Sending connection context
0620 07:10:04.384766 (+ 1170us) client_negotiation.cc:240] Negotiation successful
0620 07:10:04.385087 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":271,"spinlock_wait_cycles":41344,"thread_start_us":347,"threads_started":1}
I20260620 07:10:04.386242 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.367715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:38109 (local address 127.0.0.1:56376)
0620 07:10:04.368269 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:04.368287 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:04.368425 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:04.368739 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:04.368743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:04.369275 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:04.369616 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.369631 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.371164 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.371171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.373151 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.373167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.374468 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.374473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.374737 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.375729 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:04.375747 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:04.377118 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:04.381539 (+ 4421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:04.381544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:04.381548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:04.382091 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:04.382505 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:04.382510 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:04.382513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:04.382643 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:04.383298 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:04.383302 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:04.383581 (+ 279us) client_negotiation.cc:769] Sending connection context
0620 07:10:04.385196 (+ 1615us) client_negotiation.cc:240] Negotiation successful
0620 07:10:04.385480 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":286,"spinlock_wait_cycles":3712,"thread_start_us":293,"threads_started":1}
WARNING: no policy specified for oryx/127.6.53.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.53.124@KRBTEST.COM" created.
I20260620 07:10:04.402823 13045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:10:04.403141 13045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } }
I20260620 07:10:04.404647 13045 leader_election.cc:288] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f0bb53d5db24273b564b44d83a922bc (127.6.53.126:41085)
I20260620 07:10:04.405565 12521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0963332cf16b489e9eaf2607296f0e60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f0bb53d5db24273b564b44d83a922bc" is_pre_election: true
Jun 20 07:10:04 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939402, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.126@KRBTEST.COM for oryx/127.6.53.125@KRBTEST.COM
I20260620 07:10:04.406863 12990 leader_election.cc:302] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0963332cf16b489e9eaf2607296f0e60; no voters: 8f0bb53d5db24273b564b44d83a922bc
I20260620 07:10:04.407533 13045 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 07:10:04.414148 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.392207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56392 (local address 127.6.53.125:38109)
0620 07:10:04.392442 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:04.392452 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:04.396375 (+ 3923us) server_negotiation.cc:407] Connection header received
0620 07:10:04.396682 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:04.396686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:04.396782 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:04.396865 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:04.398355 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.399896 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.401921 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.402844 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.403122 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.408242 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:04.408250 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:04.408256 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:04.408301 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:04.411336 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.412067 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.412073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.412077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.412236 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.412629 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.412635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.412638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.412911 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:04.413072 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:04.413540 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 07:10:04.413761 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:10:04.414204 13065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.392175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:38109 (local address 127.0.0.1:56392)
0620 07:10:04.396201 (+ 4026us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:04.396256 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:04.396457 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:04.396961 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:04.396969 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:04.397860 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:04.398137 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.398151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.400068 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.400074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.401743 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.401760 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.403452 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.403462 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.403719 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.404530 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:04.404557 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:04.407998 (+ 3441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:04.411483 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:04.411498 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:04.411507 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:04.411941 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:04.412352 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:04.412357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:04.412361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:04.412519 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:04.413067 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:04.413075 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:04.413246 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 07:10:04.413440 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 07:10:04.413771 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3770,"thread_start_us":3623,"threads_started":1}
I20260620 07:10:04.416592 12511 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.53.125:38109: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0963332cf16b489e9eaf2607296f0e60"
member_type: NON_VOTER
last_known_addr {
host: "127.6.53.125"
port: 38109
}
attrs {
promote: true
}
}
dest_uuid: "8f0bb53d5db24273b564b44d83a922bc"
cas_config_opid_index: -1
I20260620 07:10:04.417627 12511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } }
I20260620 07:10:04.423931 13068 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0963332cf16b489e9eaf2607296f0e60 (127.6.53.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } } }
I20260620 07:10:04.425189 13014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:04.428362 13068 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.53.125:38109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.53.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.124.keytab.
Entry for principal oryx/127.6.53.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.124.keytab.
I20260620 07:10:04.428527 13014 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 FOLLOWER]: Refusing update from remote peer 8f0bb53d5db24273b564b44d83a922bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 07:10:04.429239 12972 master.cc:537] Master@127.6.53.125:38109 shutting down...
I20260620 07:10:04.429344 13069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } } }
I20260620 07:10:04.429661 13069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:04.430580 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.124:36739
--webserver_interface=127.6.53.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.124.keytab
--principal=oryx/127.6.53.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:41085,127.6.53.125:38109,127.6.53.124:36739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:10:04.431159 13068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:10:04.443054 12497 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 07:10:04.443114 12972 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:10:04.444288 12972 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:04.444533 12972 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: stopping tablet replica
I20260620 07:10:04.450910 12972 master.cc:552] Master@127.6.53.125:38109 shutdown complete.
I20260620 07:10:04.451097 12972 master_runner.cc:305] Clearing existing system tablet
I20260620 07:10:04.451651 12972 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:04.451838 12972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260620 07:10:04.453537 12497 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc -> Peer 0963332cf16b489e9eaf2607296f0e60 (127.6.53.125:38109): Couldn't send request to peer 0963332cf16b489e9eaf2607296f0e60. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260620 07:10:04.455752 12972 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 07:10:04.457614 13075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:04.457985 12972 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:04.458333 12972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:10:04.458434 12972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/wal
uuid: "0963332cf16b489e9eaf2607296f0e60"
format_stamp: "Formatted at 2026-06-20 07:10:04 on dist-test-slave-4n6h"
server_key: "9758648b5a53df4b71becef9ebe2b028"
server_key_iv: "633654aa52feb5a8894dafea70ca9046"
server_key_version: "encryptionkey@0"
I20260620 07:10:04.464005 12972 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:04.470947 12972 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 07:10:04.471170 12972 log.cc:1196] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 07:10:04.471665 12972 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: Deleting consensus metadata
I20260620 07:10:04.471927 12972 master_runner.cc:315] Copying system tablet from 127.6.53.126:41085
I20260620 07:10:04.474339 12972 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: tablet copy: Beginning tablet copy session from remote peer at address 127.6.53.126:41085
I20260620 07:10:04.493484 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.475176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39526 (local address 127.6.53.126:41085)
0620 07:10:04.475358 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:04.475366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:04.478798 (+ 3432us) server_negotiation.cc:407] Connection header received
0620 07:10:04.478958 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:04.478965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:04.479077 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:04.479187 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:04.480427 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.481570 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.483361 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.484024 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.484175 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.486377 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:04.486386 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:04.486392 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:04.486437 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:04.490176 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.491041 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.491049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.491053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.491219 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.491772 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.491778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.491782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.492184 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:04.492360 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:04.492819 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 07:10:04.493071 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:10:04.493522 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.475086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.0.0.1:39526)
0620 07:10:04.478652 (+ 3566us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:04.478677 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:04.478835 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:04.479233 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:04.479238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:04.479758 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:04.480228 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.480243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.481743 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.481748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.483197 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.483212 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.484200 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.484205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.484373 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.485125 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:04.485147 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:04.486114 (+ 967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:04.490383 (+ 4269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:04.490388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:04.490392 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:04.490898 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:04.491380 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:04.491388 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:04.491393 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:04.491628 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:04.492343 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:04.492348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:04.492534 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 07:10:04.492695 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 07:10:04.493059 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3429,"thread_start_us":3319,"threads_started":1}
I20260620 07:10:04.494907 12531 tablet_copy_service.cc:140] P 8f0bb53d5db24273b564b44d83a922bc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0963332cf16b489e9eaf2607296f0e60 ({username='oryx', principal='oryx/127.6.53.125@KRBTEST.COM'} at 127.0.0.1:39526)
I20260620 07:10:04.495249 12531 tablet_copy_service.cc:161] P 8f0bb53d5db24273b564b44d83a922bc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0963332cf16b489e9eaf2607296f0e60 at {username='oryx', principal='oryx/127.6.53.125@KRBTEST.COM'} at 127.0.0.1:39526: session id = 0963332cf16b489e9eaf2607296f0e60-00000000000000000000000000000000
I20260620 07:10:04.500247 12531 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 07:10:04.502983 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 07:10:04.510219 12972 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: tablet copy: Starting download of 0 data blocks...
I20260620 07:10:04.510666 12972 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: tablet copy: Starting download of 1 WAL segments...
I20260620 07:10:04.513743 12972 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 07:10:04.518525 12531 tablet_copy_service.cc:342] P 8f0bb53d5db24273b564b44d83a922bc: Request end of tablet copy session 0963332cf16b489e9eaf2607296f0e60-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.53.125@KRBTEST.COM'} at 127.0.0.1:39526
I20260620 07:10:04.518736 12531 tablet_copy_service.cc:433] P 8f0bb53d5db24273b564b44d83a922bc: ending tablet copy session 0963332cf16b489e9eaf2607296f0e60-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0963332cf16b489e9eaf2607296f0e60
I20260620 07:10:04.531589 12972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:04.534703 13082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:04.539846 13083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:04.546901 13085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:04.547809 12972 server_base.cc:1034] running on GCE node
I20260620 07:10:04.548238 12972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:04.548939 12972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:04.550194 12972 hybrid_clock.cc:648] HybridClock initialized: now 1781939404550104 us; error 55 us; skew 500 ppm
I20260620 07:10:04.550526 12972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:04 dist-test-slave-4n6h krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939404, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:04.552915 12972 init.cc:377] Logged in from keytab as oryx/127.6.53.125@KRBTEST.COM (short username oryx)
I20260620 07:10:04.554301 12972 webserver.cc:460] Webserver started at http://127.6.53.125:33547/ using document root <none> and password file <none>
I20260620 07:10:04.554651 12972 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:04.554736 12972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:04.557936 12972 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:10:04.560073 13092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:04.560554 12972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 07:10:04.560686 12972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/wal
uuid: "0963332cf16b489e9eaf2607296f0e60"
format_stamp: "Formatted at 2026-06-20 07:10:04 on dist-test-slave-4n6h"
server_key: "9758648b5a53df4b71becef9ebe2b028"
server_key_iv: "633654aa52feb5a8894dafea70ca9046"
server_key_version: "encryptionkey@0"
I20260620 07:10:04.560874 12972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:04.604707 12972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:04.609315 12972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:04.637415 12972 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.125:38109
I20260620 07:10:04.637460 13143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.125:38109 every 8 connection(s)
I20260620 07:10:04.638540 12972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-1/data/info.pb
I20260620 07:10:04.643822 13144 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:10:04.647624 13144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: Bootstrap starting.
I20260620 07:10:04.664078 13144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:10:04.664618 13144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: Bootstrap complete.
I20260620 07:10:04.665666 13144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } }
I20260620 07:10:04.665942 13144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0963332cf16b489e9eaf2607296f0e60, State: Initialized, Role: LEARNER
I20260620 07:10:04.666185 13144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } }
I20260620 07:10:04.667107 13147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } } }
I20260620 07:10:04.667419 13144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:10:04.667590 13147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: This master's current role is: LEARNER
I20260620 07:10:04.681484 13158 catalog_manager.cc:1259] Loaded cluster ID: 164f487c8df14824bf1945fe4a35bc1b
I20260620 07:10:04.681609 13158 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: loading cluster ID for follower catalog manager: success
I20260620 07:10:04.685643 13158 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: acquiring CA information for follower catalog manager: success
I20260620 07:10:04.688344 13158 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:10:04.809551 12967 heartbeater.cc:503] Master 127.6.53.126:41085 was elected leader, sending a full tablet report...
W20260620 07:10:04.846192 13072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:04.846577 13072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:04.846728 13072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:04.854048 13072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:10:04.854200 13072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:04.854327 13072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:04.854430 13072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:10:04.854506 13072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:10:04.864614 13072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:41085,127.6.53.125:38109,127.6.53.124:36739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.124.keytab
--principal=oryx/127.6.53.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.124:36739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.53.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:04.865947 13072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:04.867393 13072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:04.880009 13164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:04.880103 13166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:04.880103 13163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:04.880381 13072 server_base.cc:1034] running on GCE node
I20260620 07:10:04.881434 13072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:04.882936 13072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:04.884151 13072 hybrid_clock.cc:648] HybridClock initialized: now 1781939404884081 us; error 58 us; skew 500 ppm
I20260620 07:10:04.884543 13072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:04 dist-test-slave-4n6h krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939404, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:04.887871 13072 init.cc:377] Logged in from keytab as oryx/127.6.53.124@KRBTEST.COM (short username oryx)
I20260620 07:10:04.889313 13072 webserver.cc:460] Webserver started at http://127.6.53.124:35347/ using document root <none> and password file <none>
I20260620 07:10:04.889955 13072 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:04.890051 13072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:04.890400 13072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:04.893464 13072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/data/instance:
uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01"
format_stamp: "Formatted at 2026-06-20 07:10:04 on dist-test-slave-4n6h"
server_key: "ef1789da07b201b1ea21f33673bbc715"
server_key_iv: "f9cd0f228e6c869cb36a042fee901151"
server_key_version: "encryptionkey@0"
I20260620 07:10:04.894382 13072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/wal/instance:
uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01"
format_stamp: "Formatted at 2026-06-20 07:10:04 on dist-test-slave-4n6h"
server_key: "ef1789da07b201b1ea21f33673bbc715"
server_key_iv: "f9cd0f228e6c869cb36a042fee901151"
server_key_version: "encryptionkey@0"
I20260620 07:10:04.900882 13072 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 07:10:04.901547 13065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.893693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:38109 (local address 127.0.0.1:56406)
0620 07:10:04.893940 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:04.893964 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:04.894117 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:04.895183 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:04.895195 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:04.895222 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:04.895403 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.895411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.897434 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.897439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.899193 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.899215 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.900657 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.900662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.900934 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.900994 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 07:10:04.901125 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 07:10:04.901315 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 07:10:04.901659 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.894021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.6.53.125:38109)
0620 07:10:04.894727 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:04.894733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:04.894757 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:10:04.894839 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:04.894844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:04.894936 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:04.895064 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:04.895594 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.897247 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.899385 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.900460 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.900605 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.900862 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:04.901169 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 07:10:04.901317 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":390,"threads_started":1}
I20260620 07:10:04.902683 13119 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 07:10:04.903229 13147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f0bb53d5db24273b564b44d83a922bc. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } } }
I20260620 07:10:04.903465 13147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: This master's current role is: LEARNER
I20260620 07:10:04.905098 13174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:04.906538 13072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:10:04.906672 13072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/wal
uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01"
format_stamp: "Formatted at 2026-06-20 07:10:04 on dist-test-slave-4n6h"
server_key: "ef1789da07b201b1ea21f33673bbc715"
server_key_iv: "f9cd0f228e6c869cb36a042fee901151"
server_key_version: "encryptionkey@0"
I20260620 07:10:04.906829 13072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:04.928629 13072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:04.932919 13072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:04.933139 13072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:04.951346 13072 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.124:36739
I20260620 07:10:04.951376 13225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.124:36739 every 8 connection(s)
I20260620 07:10:04.952636 13072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/data/info.pb
I20260620 07:10:04.953576 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 13072
I20260620 07:10:04.957197 13226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:04.963889 13226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:10:04 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.53.124@KRBTEST.COM
Jun 20 07:10:04 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939404, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.124@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM
I20260620 07:10:04.976768 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.955778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.6.53.124:36739)
0620 07:10:04.956473 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:04.956485 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:04.956523 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:10:04.958140 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:04.958159 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:04.958576 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:04.958941 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:04.960356 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.961823 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.963167 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.963799 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.964038 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.968968 (+ 4930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:04.968978 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:04.968990 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:04.969036 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:04.972647 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.973462 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.973470 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.973475 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.973633 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.974032 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.974035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.974037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.974682 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:04.974983 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:04.975387 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 07:10:04.975741 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":302,"threads_started":1}
I20260620 07:10:04.980870 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.966123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.0.0.1:39538)
0620 07:10:04.966919 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:04.966973 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:04.967115 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:04.967484 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:04.967489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:04.968286 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:04.968635 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.968648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.970068 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.970071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.971389 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.971410 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.972355 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.972359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.972478 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.973049 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:04.973073 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:04.975689 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:04.978600 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:04.978608 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:04.978614 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:04.979159 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:04.979562 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:04.979566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:04.979568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:04.979678 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:04.980139 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:04.980144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:04.980269 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 07:10:04.980404 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 07:10:04.980613 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":444,"threads_started":1}
I20260620 07:10:04.981072 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.966214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.6.53.126:41085)
0620 07:10:04.966444 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:04.966449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:04.967067 (+ 618us) server_negotiation.cc:407] Connection header received
0620 07:10:04.967247 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:04.967252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:04.967327 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:04.967434 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:04.968779 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.969892 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.971549 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.972177 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.972312 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.975847 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:04.975852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:04.975855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:04.975884 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:04.978486 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.979296 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.979302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.979308 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.979466 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.979776 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.979779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.979781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.980030 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:04.980135 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:04.980547 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 07:10:04.980717 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 07:10:04.982136 13226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:10:04 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939404, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.124@KRBTEST.COM for oryx/127.6.53.125@KRBTEST.COM
I20260620 07:10:04.993674 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.982731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:38109 (local address 127.0.0.1:56422)
0620 07:10:04.982874 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:04.982887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:04.982982 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:04.983363 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:04.983367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:04.983704 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:04.983827 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.983834 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.984797 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.984805 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.985725 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.985735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.986521 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.986527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.986710 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.987253 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:04.987269 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:04.989135 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:04.991815 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:04.991818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:04.991821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:04.992166 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:04.992488 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:04.992491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:04.992494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:04.992631 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:04.993053 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:04.993056 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:04.993163 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 07:10:04.993265 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 07:10:04.993464 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 07:10:04.993752 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.982916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56422 (local address 127.6.53.125:38109)
0620 07:10:04.983119 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:04.983124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:04.983137 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:10:04.983182 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:04.983185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:04.983240 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:04.983349 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:04.983998 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.984672 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.985869 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.986366 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.986500 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.989295 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:04.989300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:04.989303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:04.989335 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:04.991705 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.992261 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.992264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.992266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.992386 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:04.992716 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:04.992719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:04.992721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:04.992963 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:04.993066 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:04.993342 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 07:10:04.993500 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:10:04.994637 13226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:10:04 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939404, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.124@KRBTEST.COM for oryx/127.6.53.124@KRBTEST.COM
I20260620 07:10:05.006121 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.995227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36739 (local address 127.0.0.1:45468)
0620 07:10:04.995460 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:04.995475 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:04.995591 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:04.995837 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:04.995840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:04.996129 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:04.996264 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.996271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.997142 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.997145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.997933 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:04.997942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.998765 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.998770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:04.998922 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:04.999456 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:04.999472 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:05.001393 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:05.004095 (+ 2702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.004099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.004102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.004489 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.004804 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.004807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.004809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.004961 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.005383 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:05.005386 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:05.005508 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 07:10:05.005612 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 07:10:05.005793 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 07:10:05.006309 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:04.995436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.6.53.124:36739)
0620 07:10:04.995583 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:04.995589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:04.995609 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:10:04.995685 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:04.995688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:04.995735 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:04.995805 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:04.996375 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.997041 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.998080 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:04.998634 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:04.998725 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.001543 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:05.001547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:05.001550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:05.001576 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:05.003959 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.004597 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.004600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.004602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.004711 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.005050 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.005052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.005054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.005290 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:05.005391 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:05.005688 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 07:10:05.005941 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":43}
I20260620 07:10:05.011202 13226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: Bootstrap starting.
I20260620 07:10:05.014518 13226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:05.015607 13226 log.cc:824] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:05.018508 13226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: No bootstrap required, opened a new log
I20260620 07:10:05.025784 13226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } }
I20260620 07:10:05.026134 13226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:05.026305 13226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc80693e6e44dc9a0e14b8bc5eb4b01, State: Initialized, Role: FOLLOWER
I20260620 07:10:05.026926 13226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } }
I20260620 07:10:05.027827 13232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } } }
I20260620 07:10:05.028115 13232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:10:05.028853 13226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 07:10:05.037963 13243 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:10:05.038092 13243 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:10:05.041555 13072 master_runner.cc:428] Detected that this master 6fc80693e6e44dc9a0e14b8bc5eb4b01 is joining an existing cluster
I20260620 07:10:05.041689 13072 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.53.124:36739
I20260620 07:10:05.058929 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.045389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39546 (local address 127.6.53.126:41085)
0620 07:10:05.045570 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:05.045582 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:05.046131 (+ 549us) server_negotiation.cc:407] Connection header received
0620 07:10:05.046381 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:05.046386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:05.046467 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:05.046543 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:05.047677 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.048401 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.049760 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.050183 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.050341 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.052205 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:05.052210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:05.052213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:05.052248 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:05.055747 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.056569 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.056577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.056581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.056743 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.057225 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.057230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.057234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.057575 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:05.057698 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:05.058344 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 07:10:05.058562 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:10:05.060621 13249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.045420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.0.0.1:39546)
0620 07:10:05.046031 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:05.046046 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:05.046191 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:05.046738 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:05.046742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:05.047245 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:05.047533 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.047543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.048549 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.048556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.049612 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.049624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.050401 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.050409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.050570 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.051186 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:05.051203 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:05.052008 (+ 805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:05.055896 (+ 3888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.055901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.055905 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.056417 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.056886 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.056893 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.056901 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.057090 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.057797 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:05.057802 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:05.058055 (+ 253us) client_negotiation.cc:769] Sending connection context
0620 07:10:05.059734 (+ 1679us) client_negotiation.cc:240] Negotiation successful
0620 07:10:05.060110 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":328,"threads_started":1}
I20260620 07:10:05.068154 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.046341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56428 (local address 127.6.53.125:38109)
0620 07:10:05.046538 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:05.046543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:05.053022 (+ 6479us) server_negotiation.cc:407] Connection header received
0620 07:10:05.053176 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:05.053182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:05.053272 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:05.053377 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:05.054963 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.056055 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.058118 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.058762 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.058854 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.061349 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:05.061355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:05.061362 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:05.061401 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:05.064533 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.065707 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.065712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.065715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.065867 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.066407 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.066413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.066416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.066779 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:05.066891 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:05.067564 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 07:10:05.067797 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:10:05.069469 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.046343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:38109 (local address 127.0.0.1:56428)
0620 07:10:05.052893 (+ 6550us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:05.052914 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:05.053045 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:05.053588 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:05.053593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:05.054379 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:05.054767 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.054778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.056248 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.056256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.057952 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.057969 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.059144 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.059150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.059299 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.059940 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:05.059961 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:05.061167 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:05.065022 (+ 3855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.065027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.065032 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.065555 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.066028 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.066032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.066035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.066272 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.067026 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:05.067035 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:05.067220 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 07:10:05.068407 (+ 1187us) client_negotiation.cc:240] Negotiation successful
0620 07:10:05.068823 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6409,"spinlock_wait_cycles":24320,"thread_start_us":6784,"threads_started":1}
I20260620 07:10:05.075616 13251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.053967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36739 (local address 127.0.0.1:45476)
0620 07:10:05.060537 (+ 6570us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:05.060560 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:05.060906 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:05.061375 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:05.061382 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:05.061965 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:05.062334 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.062345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.063842 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.063847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.065389 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.065402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.066999 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.067005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.067163 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.067838 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:05.067866 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:05.069001 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:05.073040 (+ 4039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.073047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.073051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.073438 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.073831 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.073838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.073841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.073964 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.074542 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:05.074552 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:05.074701 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 07:10:05.074833 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 07:10:05.075115 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6403,"thread_start_us":6677,"threads_started":1}
I20260620 07:10:05.075639 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.053653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45476 (local address 127.6.53.124:36739)
0620 07:10:05.053849 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:05.053855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:05.060635 (+ 6780us) server_negotiation.cc:407] Connection header received
0620 07:10:05.061080 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:05.061085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:05.061149 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:05.061212 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:05.062490 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.063645 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.065559 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.066291 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.066383 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.069178 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:05.069184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:05.069189 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:05.069220 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:05.072914 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.073560 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.073566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.073571 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.073708 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.074073 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.074077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.074081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.074416 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:05.074589 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:05.074924 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 07:10:05.075241 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":69}
I20260620 07:10:05.082875 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 12572
Jun 20 07:10:05 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.53.65@KRBTEST.COM
Jun 20 07:10:05 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939402, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.126@KRBTEST.COM for oryx/127.6.53.124@KRBTEST.COM
I20260620 07:10:05.101922 13232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:10:05.102049 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.083889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47724 (local address 127.6.53.65:42627)
0620 07:10:05.084468 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:05.084476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:05.084499 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:10:05.084554 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:05.084560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:05.084765 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:05.084930 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:05.086176 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.087240 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.088633 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.089189 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.089333 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.094033 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:05.094046 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:05.094061 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:05.094098 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:05.098355 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.099177 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.099188 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.099192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.099389 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.099978 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.099983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.099987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.100450 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:05.100815 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:05.101179 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 07:10:05.101462 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":286,"threads_started":1}
I20260620 07:10:05.102222 13232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } }
I20260620 07:10:05.103931 13232 leader_election.cc:288] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f0bb53d5db24273b564b44d83a922bc (127.6.53.126:41085), 0963332cf16b489e9eaf2607296f0e60 (127.6.53.125:38109)
I20260620 07:10:05.104074 12695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:05.105053 12521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f0bb53d5db24273b564b44d83a922bc" is_pre_election: true
I20260620 07:10:05.105629 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0963332cf16b489e9eaf2607296f0e60" is_pre_election: true
I20260620 07:10:05.106846 13177 leader_election.cc:302] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fc80693e6e44dc9a0e14b8bc5eb4b01; no voters: 0963332cf16b489e9eaf2607296f0e60, 8f0bb53d5db24273b564b44d83a922bc
I20260620 07:10:05.107558 13232 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 07:10:05.112123 13065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.067448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36739 (local address 127.0.0.1:45484)
0620 07:10:05.067764 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:05.067783 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:05.067885 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:05.071910 (+ 4025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:05.071916 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:05.072349 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:05.072590 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.072603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.096399 (+ 23796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.096408 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.097474 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.097491 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.098394 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.098401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.098605 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.099236 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:05.099281 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:05.102731 (+ 3450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:05.108854 (+ 6123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.108860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.108864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.109407 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.109916 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.109922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.109926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.110084 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.111076 (+ 992us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:05.111081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:05.111261 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 07:10:05.111410 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 07:10:05.111709 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260620 07:10:05.113297 13252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.068110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.6.53.124:36739)
0620 07:10:05.071527 (+ 3417us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:05.071533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:05.071554 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:10:05.071637 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:05.071642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:05.071725 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:05.071837 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:05.094715 (+ 22878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.096216 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.097654 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.098149 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.098448 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.103004 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:05.103016 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:05.103021 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:05.103063 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:05.108644 (+ 5581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.109580 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.109586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.109590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.109749 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.110282 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.110288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.110291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.110880 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:05.112497 (+ 1617us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:05.112615 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 07:10:05.112879 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3264,"thread_start_us":3129,"threads_started":1}
I20260620 07:10:05.115239 12511 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.53.124:36739: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01"
member_type: NON_VOTER
last_known_addr {
host: "127.6.53.124"
port: 36739
}
attrs {
promote: true
}
}
dest_uuid: "8f0bb53d5db24273b564b44d83a922bc"
cas_config_opid_index: 5
I20260620 07:10:05.116360 12511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } }
I20260620 07:10:05.118217 13068 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6fc80693e6e44dc9a0e14b8bc5eb4b01 (127.6.53.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } } }
I20260620 07:10:05.119414 13119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 LEARNER]: Refusing update from remote peer 8f0bb53d5db24273b564b44d83a922bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 07:10:05.121357 13201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:05.127058 13201 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 1 FOLLOWER]: Refusing update from remote peer 8f0bb53d5db24273b564b44d83a922bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 07:10:05.134613 13257 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 07:10:05.139276 13255 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:05.141824 13068 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.53.124:36739
I20260620 07:10:05.142377 13257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } } }
I20260620 07:10:05.142714 13257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:05.147241 13072 master.cc:537] Master@127.6.53.124:36739 shutting down...
I20260620 07:10:05.146754 13119 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6fc80693e6e44dc9a0e14b8bc5eb4b01 (127.6.53.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } } }
I20260620 07:10:05.153873 13201 raft_consensus.cc:858] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } }
W20260620 07:10:05.154184 13201 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } }
I20260620 07:10:05.170806 13147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } } }
I20260620 07:10:05.171169 13147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: This master's current role is: LEARNER
I20260620 07:10:05.172367 13068 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc: attempting to promote NON_VOTER 0963332cf16b489e9eaf2607296f0e60 to VOTER
I20260620 07:10:05.173269 13068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } }
I20260620 07:10:05.176440 13118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 LEARNER]: Refusing update from remote peer 8f0bb53d5db24273b564b44d83a922bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 07:10:05.177317 13255 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 07:10:05.185118 13180 tablet.cc:2369] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:10:05.189330 12497 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc -> Peer 6fc80693e6e44dc9a0e14b8bc5eb4b01 (127.6.53.124:36739): Couldn't send request to peer 6fc80693e6e44dc9a0e14b8bc5eb4b01. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 07:10:05.196136 13255 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0963332cf16b489e9eaf2607296f0e60 (127.6.53.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } } }
I20260620 07:10:05.200182 13256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } } }
I20260620 07:10:05.200580 13256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:05.201138 13256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } } }
I20260620 07:10:05.201354 13256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:05.202128 13118 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0963332cf16b489e9eaf2607296f0e60 (127.6.53.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } } }
I20260620 07:10:05.206280 13147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } } }
I20260620 07:10:05.206523 13147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:10:05.216262 13201 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0963332cf16b489e9eaf2607296f0e60 (127.6.53.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } } }
I20260620 07:10:05.227461 13263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } } }
I20260620 07:10:05.227698 13263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 07:10:05.226986 13266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } } }
I20260620 07:10:05.227916 13266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 07:10:05.228073 13232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f0bb53d5db24273b564b44d83a922bc. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: NON_VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: true } } }
I20260620 07:10:05.228250 13232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 07:10:05.229516 13072 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 07:10:05.230000 13072 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 07:10:05.230177 13072 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: stopping tablet replica
I20260620 07:10:05.237598 13072 master.cc:552] Master@127.6.53.124:36739 shutdown complete.
I20260620 07:10:05.237680 13072 master_runner.cc:305] Clearing existing system tablet
I20260620 07:10:05.238055 13072 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:05.238148 13072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:05.240896 13072 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:10:05.242631 13268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:05.243011 13072 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:05.243268 13072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260620 07:10:05.243364 13072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/wal
uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01"
format_stamp: "Formatted at 2026-06-20 07:10:04 on dist-test-slave-4n6h"
server_key: "ef1789da07b201b1ea21f33673bbc715"
server_key_iv: "f9cd0f228e6c869cb36a042fee901151"
server_key_version: "encryptionkey@0"
I20260620 07:10:05.247699 13072 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:05.254230 13072 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 07:10:05.254429 13072 log.cc:1196] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 07:10:05.254927 13072 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: Deleting consensus metadata
I20260620 07:10:05.255173 13072 master_runner.cc:315] Copying system tablet from 127.6.53.126:41085
I20260620 07:10:05.257648 13072 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: tablet copy: Beginning tablet copy session from remote peer at address 127.6.53.126:41085
I20260620 07:10:05.277712 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 12572
I20260620 07:10:05.283193 13274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.258418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.0.0.1:39552)
0620 07:10:05.266737 (+ 8319us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:05.266762 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:05.266928 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:05.267546 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:05.267553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:05.268233 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:05.268503 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.268516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.270041 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.270049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.271630 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.271645 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.272579 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.272586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.272775 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.273510 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:05.273534 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:05.274632 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:05.280118 (+ 5486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.280124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.280129 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.280676 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.281118 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.281124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.281128 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.281291 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.281943 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:05.281947 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:05.282178 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 07:10:05.282353 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 07:10:05.282748 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8156,"thread_start_us":8010,"threads_started":1}
I20260620 07:10:05.283727 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.258488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39552 (local address 127.6.53.126:41085)
0620 07:10:05.258711 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:05.258717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:05.266910 (+ 8193us) server_negotiation.cc:407] Connection header received
0620 07:10:05.267129 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:05.267136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:05.267230 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:05.267366 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:05.268685 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.269873 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.271785 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.272418 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.272540 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.274829 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:05.274835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:05.274839 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:05.274875 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:05.279981 (+ 5106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.280821 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.280830 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.280836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.280994 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.281420 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.281427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.281433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.281804 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:05.281990 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:05.282665 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 07:10:05.283026 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 07:10:05.284281 12531 tablet_copy_service.cc:140] P 8f0bb53d5db24273b564b44d83a922bc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6fc80693e6e44dc9a0e14b8bc5eb4b01 ({username='oryx', principal='oryx/127.6.53.124@KRBTEST.COM'} at 127.0.0.1:39552)
I20260620 07:10:05.284466 12531 tablet_copy_service.cc:161] P 8f0bb53d5db24273b564b44d83a922bc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6fc80693e6e44dc9a0e14b8bc5eb4b01 at {username='oryx', principal='oryx/127.6.53.124@KRBTEST.COM'} at 127.0.0.1:39552: session id = 6fc80693e6e44dc9a0e14b8bc5eb4b01-00000000000000000000000000000000
I20260620 07:10:05.287976 12531 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 07:10:05.289955 13072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 07:10:05.297848 13072 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: tablet copy: Starting download of 0 data blocks...
I20260620 07:10:05.298218 13072 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: tablet copy: Starting download of 1 WAL segments...
I20260620 07:10:05.299330 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:42627
--local_ip_for_outbound_sockets=127.6.53.65
--tserver_master_addrs=127.6.53.126:41085
--webserver_port=43633
--webserver_interface=127.6.53.65
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.65.keytab
--principal=oryx/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.53.126:41085,127.6.53.125:38109
--tserver_master_addrs=127.6.53.126:41085,127.6.53.125:38109,127.6.53.124:36739 with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:10:05.300680 13072 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 07:10:05.305562 12531 tablet_copy_service.cc:342] P 8f0bb53d5db24273b564b44d83a922bc: Request end of tablet copy session 6fc80693e6e44dc9a0e14b8bc5eb4b01-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.53.124@KRBTEST.COM'} at 127.0.0.1:39552
I20260620 07:10:05.305701 12531 tablet_copy_service.cc:433] P 8f0bb53d5db24273b564b44d83a922bc: ending tablet copy session 6fc80693e6e44dc9a0e14b8bc5eb4b01-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6fc80693e6e44dc9a0e14b8bc5eb4b01
I20260620 07:10:05.318380 13072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:05.321023 13276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:05.330647 13277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:05.340022 13279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:05.340101 13072 server_base.cc:1034] running on GCE node
I20260620 07:10:05.340638 13072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:05.341297 13072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:05.342516 13072 hybrid_clock.cc:648] HybridClock initialized: now 1781939405342454 us; error 50 us; skew 500 ppm
I20260620 07:10:05.342748 13072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:05 dist-test-slave-4n6h krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939405, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:05.344868 13072 init.cc:377] Logged in from keytab as oryx/127.6.53.124@KRBTEST.COM (short username oryx)
I20260620 07:10:05.351389 13072 webserver.cc:460] Webserver started at http://127.6.53.124:35715/ using document root <none> and password file <none>
I20260620 07:10:05.351722 13072 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:05.351824 13072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:05.354768 13072 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:10:05.356616 13286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:05.357031 13072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 07:10:05.357182 13072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/wal
uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01"
format_stamp: "Formatted at 2026-06-20 07:10:04 on dist-test-slave-4n6h"
server_key: "ef1789da07b201b1ea21f33673bbc715"
server_key_iv: "f9cd0f228e6c869cb36a042fee901151"
server_key_version: "encryptionkey@0"
I20260620 07:10:05.357411 13072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:05.377683 13072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:05.382308 13072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:05.405977 13072 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.124:36739
I20260620 07:10:05.406025 13337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.124:36739 every 8 connection(s)
I20260620 07:10:05.407202 13072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/master-2/data/info.pb
I20260620 07:10:05.412827 13338 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:10:05.416658 13338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: Bootstrap starting.
I20260620 07:10:05.432072 13338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:10:05.432614 13338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: Bootstrap complete.
I20260620 07:10:05.433768 13338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } }
I20260620 07:10:05.434134 13338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6fc80693e6e44dc9a0e14b8bc5eb4b01, State: Initialized, Role: LEARNER
I20260620 07:10:05.434492 13338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } }
I20260620 07:10:05.435391 13341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } } }
I20260620 07:10:05.435776 13338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:10:05.436055 13341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: This master's current role is: LEARNER
I20260620 07:10:05.446844 13352 catalog_manager.cc:1259] Loaded cluster ID: 164f487c8df14824bf1945fe4a35bc1b
I20260620 07:10:05.446939 13352 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: loading cluster ID for follower catalog manager: success
I20260620 07:10:05.450493 13352 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: acquiring CA information for follower catalog manager: success
I20260620 07:10:05.452881 13352 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 07:10:05.664080 13275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:05.664333 13275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:05.664516 13275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:05.671895 13275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:05.672045 13275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:05.672230 13275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:10:05.682010 13275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.65.keytab
--principal=oryx/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:42627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=43633
--tserver_master_addrs=127.6.53.126:41085,127.6.53.125:38109,127.6.53.124:36739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:05.683373 13275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:05.684680 13275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:05.698201 13358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:05.698275 13360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:05.698323 13357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:05.699718 13275 server_base.cc:1034] running on GCE node
I20260620 07:10:05.700480 13275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:05.701455 13275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:05.702736 13275 hybrid_clock.cc:648] HybridClock initialized: now 1781939405702663 us; error 57 us; skew 500 ppm
I20260620 07:10:05.703029 13275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:05 dist-test-slave-4n6h krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939405, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:05.705977 13275 init.cc:377] Logged in from keytab as oryx/127.6.53.65@KRBTEST.COM (short username oryx)
I20260620 07:10:05.707479 13275 webserver.cc:460] Webserver started at http://127.6.53.65:43633/ using document root <none> and password file <none>
I20260620 07:10:05.708158 13275 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:05.708318 13275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:05.713421 13275 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:10:05.716179 13367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:05.717617 13275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:10:05.717746 13275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "659e9bb195db4166b7c0c6bc30452fbc"
format_stamp: "Formatted at 2026-06-20 07:10:02 on dist-test-slave-4n6h"
server_key: "8a50ce9e1ce6bb9609a6ceb5bd7fa5b4"
server_key_iv: "fc29487bcd329035fdd27dfa6af70f5f"
server_key_version: "encryptionkey@0"
I20260620 07:10:05.718741 13275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:05.731040 13275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:05.734849 13275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:05.735103 13275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:05.735986 13275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:05.737303 13275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:05.737386 13275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:05.737473 13275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:05.737524 13275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:05.770655 13275 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:42627
I20260620 07:10:05.770684 13479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:42627 every 8 connection(s)
I20260620 07:10:05.772303 13275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:10:05.774950 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 13275
I20260620 07:10:05.775353 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 12706
I20260620 07:10:05.782799 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.772856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36739 (local address 127.0.0.1:45492)
0620 07:10:05.773361 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:05.773383 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:05.773539 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:05.774976 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:05.774982 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:05.775007 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:05.775192 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.775201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.776837 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.776847 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.779509 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.779534 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.781824 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.781833 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.782098 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.782146 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 07:10:05.782218 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 07:10:05.782393 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":249,"threads_started":1}
I20260620 07:10:05.784327 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.772909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.6.53.124:36739)
0620 07:10:05.774659 (+ 1750us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:05.774664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:05.774680 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:10:05.774738 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:05.774741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:05.774813 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:05.774962 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:05.775332 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.776634 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.779822 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.781569 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.783268 (+ 1699us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.783586 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:05.783652 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 07:10:05.783843 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1626,"thread_start_us":1532,"threads_started":1}
I20260620 07:10:05.785094 13313 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 07:10:05.785765 13341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f0bb53d5db24273b564b44d83a922bc. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } } }
I20260620 07:10:05.786028 13341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: This master's current role is: LEARNER
Jun 20 07:10:05 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939405, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.124@KRBTEST.COM
Jun 20 07:10:05 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939405, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM
Jun 20 07:10:05 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939405, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.125@KRBTEST.COM
Jun 20 07:10:05 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.53.66@KRBTEST.COM
I20260620 07:10:05.812804 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.782277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:59973 (local address 127.6.53.124:36739)
0620 07:10:05.784573 (+ 2296us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:05.784580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:05.787411 (+ 2831us) server_negotiation.cc:407] Connection header received
0620 07:10:05.791461 (+ 4050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:05.791466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:05.791555 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:05.791680 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:05.793789 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.794903 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.797491 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.798137 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.798242 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.804330 (+ 6088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:05.804336 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:05.804341 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:05.804380 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:05.808115 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.809869 (+ 1754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.809878 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.809882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.810062 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.810670 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.810676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.810680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.811069 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:05.811183 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:05.812159 (+ 976us) server_negotiation.cc:299] Negotiation successful
0620 07:10:05.812407 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2193,"thread_start_us":8925,"threads_started":1}
I20260620 07:10:05.815892 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.791875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:43549 (local address 127.6.53.126:41085)
0620 07:10:05.792357 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:05.792362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:05.793099 (+ 737us) server_negotiation.cc:407] Connection header received
0620 07:10:05.793308 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:05.793313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:05.793398 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:05.793493 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:05.794862 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.796093 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.798112 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.798810 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.798901 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.805378 (+ 6477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:05.805384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:05.805389 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:05.805437 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:05.809833 (+ 4396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.813638 (+ 3805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.813645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.813648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.813791 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.814324 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.814331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.814336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.814699 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:05.814849 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:05.815248 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 07:10:05.815503 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":261,"threads_started":1}
I20260620 07:10:05.816344 13489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.792402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.65:43549)
0620 07:10:05.793005 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:05.793027 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:05.793181 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:05.793707 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:05.793710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:05.794393 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:05.794709 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.794720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.796266 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.796272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.797960 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.797973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.799275 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.799284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.799492 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.800120 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:05.800153 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:05.805165 (+ 5012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:05.811403 (+ 6238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.811407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.811410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.813530 (+ 2120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.813911 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.813918 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.813925 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.814096 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.814832 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:05.814836 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:05.814978 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 07:10:05.815142 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 07:10:05.815410 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":5,"spinlock_wait_cycles":30592,"thread_start_us":295,"threads_started":1}
I20260620 07:10:05.816370 13485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.781558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36739 (local address 127.6.53.65:59973)
0620 07:10:05.787215 (+ 5657us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:05.787293 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:05.790732 (+ 3439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:05.791919 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:05.791928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:05.792707 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:05.793608 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.793634 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.795120 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.795125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.797328 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.797341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.798591 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.798599 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.798957 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.800089 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:05.800118 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:05.804077 (+ 3959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:05.808966 (+ 4889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.808977 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.808995 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.809723 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.810221 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.810226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.810232 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.810527 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.811289 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:05.811295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:05.811739 (+ 444us) client_negotiation.cc:769] Sending connection context
0620 07:10:05.814328 (+ 2589us) client_negotiation.cc:240] Negotiation successful
0620 07:10:05.814742 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5249,"thread_start_us":5024,"threads_started":1}
I20260620 07:10:05.817179 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.798295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:58031 (local address 127.6.53.125:38109)
0620 07:10:05.799047 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:05.799052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:05.799077 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:10:05.799594 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:05.799601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:05.799703 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:05.799824 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:05.801365 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.802537 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.804756 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.805399 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.805528 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.809834 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:05.809840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:05.809844 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:05.809885 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:05.813370 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.814222 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.814229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.814235 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.814410 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.814831 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.814836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.814839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.815225 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:05.815344 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:05.816617 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0620 07:10:05.816824 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":508,"threads_started":1}
I20260620 07:10:05.816439 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.797719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:38109 (local address 127.6.53.65:58031)
0620 07:10:05.798462 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:05.798482 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:05.798593 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:05.799883 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:05.799889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:05.800894 (+ 1005us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:05.801228 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.801240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.802691 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.802696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.804632 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:05.804645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.805786 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.805793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:05.805969 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.806638 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:05.806665 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:05.809599 (+ 2934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:05.813531 (+ 3932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.813539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.813543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.814081 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.814546 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:05.814550 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:05.814554 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:05.814708 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:05.815415 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:05.815420 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:05.815546 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 07:10:05.815713 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 07:10:05.815993 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":757,"threads_started":1}
I20260620 07:10:05.818030 13487 heartbeater.cc:347] Connected to a master server at 127.6.53.126:41085
I20260620 07:10:05.818387 13487 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:05.818725 13490 heartbeater.cc:347] Connected to a master server at 127.6.53.125:38109
I20260620 07:10:05.818893 13490 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:05.819456 13480 heartbeater.cc:347] Connected to a master server at 127.6.53.124:36739
I20260620 07:10:05.819483 13490 heartbeater.cc:511] Master 127.6.53.125:38109 requested a full tablet report, sending...
I20260620 07:10:05.819597 13480 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:05.819952 13480 heartbeater.cc:511] Master 127.6.53.124:36739 requested a full tablet report, sending...
I20260620 07:10:05.819507 13487 heartbeater.cc:511] Master 127.6.53.126:41085 requested a full tablet report, sending...
I20260620 07:10:05.821345 12511 ts_manager.cc:194] Re-registered known tserver with Master: 659e9bb195db4166b7c0c6bc30452fbc (127.6.53.65:42627)
I20260620 07:10:05.822047 13109 ts_manager.cc:194] Registered new tserver with Master: 659e9bb195db4166b7c0c6bc30452fbc (127.6.53.65:42627)
I20260620 07:10:05.822023 13303 ts_manager.cc:194] Registered new tserver with Master: 659e9bb195db4166b7c0c6bc30452fbc (127.6.53.65:42627)
I20260620 07:10:05.822360 12511 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:43549
I20260620 07:10:05.824349 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:05.776538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.6.53.66:39055)
0620 07:10:05.795950 (+ 19412us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:05.795962 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:05.796000 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:10:05.796081 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:05.796093 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:05.796396 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:05.801539 (+ 5143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:05.801774 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.803417 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.805638 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:05.806601 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:05.807346 (+ 745us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:05.812966 (+ 5620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:05.812981 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:05.812999 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:05.813055 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:05.819347 (+ 6292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.820372 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.820381 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.820384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.820567 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:05.822526 (+ 1959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:05.822531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:05.822534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:05.822984 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:05.823347 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:05.823723 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 07:10:05.823990 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19191,"thread_start_us":20015,"threads_started":1}
I20260620 07:10:05.825472 12829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:05.910615 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 12706
I20260620 07:10:05.926911 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:39055
--local_ip_for_outbound_sockets=127.6.53.66
--tserver_master_addrs=127.6.53.126:41085
--webserver_port=42005
--webserver_interface=127.6.53.66
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.66.keytab
--principal=oryx/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.53.126:41085,127.6.53.125:38109
--tserver_master_addrs=127.6.53.126:41085,127.6.53.125:38109,127.6.53.124:36739 with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:10:06.161060 13257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: NON_VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: true } } }
I20260620 07:10:06.161397 13257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:06.165696 13257 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc: attempting to promote NON_VOTER 6fc80693e6e44dc9a0e14b8bc5eb4b01 to VOTER
I20260620 07:10:06.166572 13257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } }
I20260620 07:10:06.168876 13119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 FOLLOWER]: Refusing update from remote peer 8f0bb53d5db24273b564b44d83a922bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 07:10:06.168883 13313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 1 LEARNER]: Refusing update from remote peer 8f0bb53d5db24273b564b44d83a922bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 07:10:06.169550 13257 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 07:10:06.169973 13496 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 07:10:06.172653 13497 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6fc80693e6e44dc9a0e14b8bc5eb4b01 (127.6.53.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } } }
I20260620 07:10:06.173210 13119 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6fc80693e6e44dc9a0e14b8bc5eb4b01 (127.6.53.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } } }
I20260620 07:10:06.176668 13497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } } }
I20260620 07:10:06.177006 13497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:06.177265 13497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } } }
I20260620 07:10:06.177556 13497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:06.181456 13500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } } }
I20260620 07:10:06.181895 13500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:10:06.185189 13501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } } }
I20260620 07:10:06.185606 13501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:06.185915 13313 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6fc80693e6e44dc9a0e14b8bc5eb4b01 (127.6.53.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } } }
I20260620 07:10:06.190038 13341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f0bb53d5db24273b564b44d83a922bc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } } }
I20260620 07:10:06.190361 13341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 07:10:06.257438 13495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:06.257781 13495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:06.257876 13495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:06.265292 13495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:06.265396 13495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:06.265509 13495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:10:06.275455 13495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.66.keytab
--principal=oryx/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:39055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=42005
--tserver_master_addrs=127.6.53.126:41085,127.6.53.125:38109,127.6.53.124:36739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:06.276839 13495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:06.278074 13495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:06.291715 13507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:06.291687 13509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:06.291886 13506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:06.292936 13495 server_base.cc:1034] running on GCE node
I20260620 07:10:06.293481 13495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:06.294384 13495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:06.295607 13495 hybrid_clock.cc:648] HybridClock initialized: now 1781939406295556 us; error 35 us; skew 500 ppm
I20260620 07:10:06.295960 13495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:06 dist-test-slave-4n6h krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:06.299113 13495 init.cc:377] Logged in from keytab as oryx/127.6.53.66@KRBTEST.COM (short username oryx)
I20260620 07:10:06.300441 13495 webserver.cc:460] Webserver started at http://127.6.53.66:42005/ using document root <none> and password file <none>
I20260620 07:10:06.301173 13495 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:06.301327 13495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:06.306435 13495 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:10:06.308912 13516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:06.310179 13495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:10:06.310377 13495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "42141b2ca9ed4593a530955665d7a0fc"
format_stamp: "Formatted at 2026-06-20 07:10:03 on dist-test-slave-4n6h"
server_key: "e703b6298ccb4e2ed5fe96a9c45bb754"
server_key_iv: "b98791a764e9049c95bb4b1752a05b06"
server_key_version: "encryptionkey@0"
I20260620 07:10:06.311323 13495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:06.328852 13495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:06.334709 13495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:06.335153 13495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:06.336464 13495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:06.338081 13495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:06.338217 13495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:06.338379 13495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:06.338486 13495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:06.387313 13495 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:39055
I20260620 07:10:06.387409 13628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:39055 every 8 connection(s)
I20260620 07:10:06.388821 13495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:10:06.398044 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 13495
I20260620 07:10:06.398530 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 12839
Jun 20 07:10:06 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.124@KRBTEST.COM
Jun 20 07:10:06 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM
I20260620 07:10:06.428532 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:06.396451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:50869 (local address 127.6.53.126:41085)
0620 07:10:06.411548 (+ 15097us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:06.411554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:06.411577 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:10:06.411639 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:06.411644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:06.411737 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:06.411862 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:06.413142 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:06.414457 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:06.415896 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:06.416819 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:06.416964 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:06.420900 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:06.420908 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:06.420912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:06.420950 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:06.424834 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:06.425636 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:06.425642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:06.425647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:06.425801 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:06.426195 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:06.426199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:06.426203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:06.426718 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:06.426865 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:06.427905 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0620 07:10:06.428138 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14886,"thread_start_us":15513,"threads_started":1}
I20260620 07:10:06.430173 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:06.396202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.66:50869)
0620 07:10:06.397103 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:06.397124 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:06.400351 (+ 3227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:06.411872 (+ 11521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:06.411877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:06.412386 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:06.412594 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:06.412603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:06.414582 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:06.414590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:06.415730 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:06.415741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:06.417013 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:06.417019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:06.417248 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:06.417867 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:06.417894 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:06.420706 (+ 2812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:06.425178 (+ 4472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:06.425181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:06.425184 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:06.425505 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:06.425943 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:06.425952 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:06.425954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:06.426059 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:06.427112 (+ 1053us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:06.427119 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:06.427469 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 07:10:06.427827 (+ 358us) client_negotiation.cc:240] Negotiation successful
0620 07:10:06.428361 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":518,"threads_started":1}
I20260620 07:10:06.431981 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:06.393067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:36333 (local address 127.6.53.124:36739)
0620 07:10:06.393693 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:06.393700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:06.395379 (+ 1679us) server_negotiation.cc:407] Connection header received
0620 07:10:06.400872 (+ 5493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:06.400881 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:06.401049 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:06.401159 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:06.404544 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:06.406321 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:06.409704 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:06.410741 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:06.410868 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:06.419925 (+ 9057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:06.419930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:06.419934 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:06.420003 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:06.423843 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:06.424915 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:06.424921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:06.424925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:06.425074 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:06.426399 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:06.426405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:06.426407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:06.426884 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:06.427038 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:06.431365 (+ 4327us) server_negotiation.cc:299] Negotiation successful
0620 07:10:06.431662 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":308,"threads_started":1}
I20260620 07:10:06.432910 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:06.393855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36739 (local address 127.6.53.66:36333)
0620 07:10:06.395150 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:06.395203 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:06.399967 (+ 4764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:06.401844 (+ 1877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:06.401864 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:06.403065 (+ 1201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:06.404233 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:06.404268 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:06.406618 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:06.406627 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:06.409477 (+ 2850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:06.409499 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:06.411342 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:06.411348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:06.411691 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:06.413652 (+ 1961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:06.413691 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:06.419683 (+ 5992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:06.424121 (+ 4438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:06.424131 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:06.424151 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:06.424766 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:06.425680 (+ 914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:06.425687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:06.425692 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:06.426269 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:06.430672 (+ 4403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:06.430677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:06.430945 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 07:10:06.432271 (+ 1326us) client_negotiation.cc:240] Negotiation successful
0620 07:10:06.432606 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"thread_start_us":538,"threads_started":1}
I20260620 07:10:06.436765 13630 heartbeater.cc:347] Connected to a master server at 127.6.53.126:41085
I20260620 07:10:06.437096 13630 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:06.438302 13630 heartbeater.cc:511] Master 127.6.53.126:41085 requested a full tablet report, sending...
I20260620 07:10:06.439925 13629 heartbeater.cc:347] Connected to a master server at 127.6.53.124:36739
I20260620 07:10:06.440097 13629 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:06.440641 13629 heartbeater.cc:511] Master 127.6.53.124:36739 requested a full tablet report, sending...
I20260620 07:10:06.442520 12511 ts_manager.cc:194] Re-registered known tserver with Master: 42141b2ca9ed4593a530955665d7a0fc (127.6.53.66:39055)
I20260620 07:10:06.442673 13303 ts_manager.cc:194] Registered new tserver with Master: 42141b2ca9ed4593a530955665d7a0fc (127.6.53.66:39055)
I20260620 07:10:06.443472 12511 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:50869
Jun 20 07:10:06 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939406, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.125@KRBTEST.COM
Jun 20 07:10:06 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.53.67@KRBTEST.COM
I20260620 07:10:06.456048 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:06.399322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:38109 (local address 127.6.53.66:38863)
0620 07:10:06.431191 (+ 31869us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:06.431216 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:06.431325 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:06.437443 (+ 6118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:06.437452 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:06.438037 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:06.438298 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:06.438315 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:06.440366 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:06.440372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:06.442223 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:06.442239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:06.443365 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:06.443372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:06.443588 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:06.444455 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:06.444483 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:06.447840 (+ 3357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:06.452702 (+ 4862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:06.452707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:06.452712 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:06.453218 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:06.454037 (+ 819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:06.454042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:06.454046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:06.454212 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:06.454930 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:06.454938 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:06.455141 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 07:10:06.455299 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 07:10:06.455675 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31703,"mutex_wait_us":4,"thread_start_us":35307,"threads_started":1}
I20260620 07:10:06.456470 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:06.397336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:38863 (local address 127.6.53.125:38109)
0620 07:10:06.434804 (+ 37468us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:06.434811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:06.434836 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:10:06.434918 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:06.434927 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:06.435038 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:06.438480 (+ 3442us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:06.438682 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:06.440100 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:06.442510 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:06.443184 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:06.443333 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:06.448049 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:06.448057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:06.448064 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:06.448133 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:06.452556 (+ 4423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:06.453544 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:06.453552 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:06.453556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:06.453896 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:06.454445 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:06.454450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:06.454452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:06.454781 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:06.454946 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:06.455632 (+ 686us) server_negotiation.cc:299] Negotiation successful
0620 07:10:06.455995 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37273,"thread_start_us":38385,"threads_started":1}
I20260620 07:10:06.457485 13631 heartbeater.cc:347] Connected to a master server at 127.6.53.125:38109
I20260620 07:10:06.457630 13631 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:06.458098 13631 heartbeater.cc:511] Master 127.6.53.125:38109 requested a full tablet report, sending...
I20260620 07:10:06.460037 13109 ts_manager.cc:194] Registered new tserver with Master: 42141b2ca9ed4593a530955665d7a0fc (127.6.53.66:39055)
I20260620 07:10:06.462963 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:06.400883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43756 (local address 127.6.53.67:42697)
0620 07:10:06.436491 (+ 35608us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:06.436500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:06.437148 (+ 648us) server_negotiation.cc:407] Connection header received
0620 07:10:06.437383 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:06.437393 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:06.437711 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:06.437913 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:06.440664 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:06.444535 (+ 3871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:06.445979 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:06.446998 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:06.447151 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:06.453652 (+ 6501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:06.453666 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:06.453682 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:06.453720 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:06.458083 (+ 4363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:06.460258 (+ 2175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:06.460275 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:06.460277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:06.460485 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:06.461044 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:06.461049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:06.461051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:06.461424 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:06.461872 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:06.462370 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 07:10:06.462646 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35385,"thread_start_us":35166,"threads_started":1}
I20260620 07:10:06.464557 12962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:06.563998 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 12839
I20260620 07:10:06.583844 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:42697
--local_ip_for_outbound_sockets=127.6.53.67
--tserver_master_addrs=127.6.53.126:41085
--webserver_port=45071
--webserver_interface=127.6.53.67
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.67.keytab
--principal=oryx/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.53.126:41085,127.6.53.125:38109
--tserver_master_addrs=127.6.53.126:41085,127.6.53.125:38109,127.6.53.124:36739 with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:10:06.824898 13487 heartbeater.cc:503] Master 127.6.53.126:41085 was elected leader, sending a full tablet report...
W20260620 07:10:06.987336 13642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:06.987608 13642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:06.987707 13642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:06.994973 13642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:06.995126 13642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:06.995713 13642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:10:07.006295 13642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.67.keytab
--principal=oryx/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:42697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=45071
--tserver_master_addrs=127.6.53.126:41085,127.6.53.125:38109,127.6.53.124:36739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:07.007875 13642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:07.009377 13642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:07.025177 13648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:07.025290 13650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:07.025908 13647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:07.026508 13642 server_base.cc:1034] running on GCE node
I20260620 07:10:07.027257 13642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:07.028542 13642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:07.030016 13642 hybrid_clock.cc:648] HybridClock initialized: now 1781939407029949 us; error 53 us; skew 500 ppm
I20260620 07:10:07.030483 13642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:07 dist-test-slave-4n6h krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939407, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:07.034159 13642 init.cc:377] Logged in from keytab as oryx/127.6.53.67@KRBTEST.COM (short username oryx)
I20260620 07:10:07.036803 13642 webserver.cc:460] Webserver started at http://127.6.53.67:45071/ using document root <none> and password file <none>
I20260620 07:10:07.037796 13642 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:07.037992 13642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:07.045356 13642 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:10:07.049007 13657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:07.050735 13642 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 07:10:07.050876 13642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "ca2ec70afce24c1099d2f07d015010ec"
format_stamp: "Formatted at 2026-06-20 07:10:03 on dist-test-slave-4n6h"
server_key: "7c75f98c9c53bb73b41e976990882d69"
server_key_iv: "ad62f4544c8de45bc96b3501ce3145fc"
server_key_version: "encryptionkey@0"
I20260620 07:10:07.051887 13642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:07.064894 13642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:07.069412 13642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:07.069653 13642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:07.070782 13642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:07.072485 13642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:07.072576 13642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:07.072675 13642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:07.072758 13642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:07.118216 13642 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:42697
I20260620 07:10:07.118350 13769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:42697 every 8 connection(s)
I20260620 07:10:07.119809 13642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:10:07.123566 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 13642
Jun 20 07:10:07 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939407, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM
Jun 20 07:10:07 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939407, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.124@KRBTEST.COM
I20260620 07:10:07.163115 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.125983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:48815 (local address 127.6.53.126:41085)
0620 07:10:07.126576 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:07.126585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:07.127373 (+ 788us) server_negotiation.cc:407] Connection header received
0620 07:10:07.128741 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:07.128751 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:07.128875 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:07.128979 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:07.137565 (+ 8586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.138767 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.141626 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.142275 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.142382 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.148179 (+ 5797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:07.148186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:07.148190 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:07.148231 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:07.153482 (+ 5251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.154477 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.154483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.154487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.154628 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.155155 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.155161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.155165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.155549 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:07.155669 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:07.162383 (+ 6714us) server_negotiation.cc:299] Negotiation successful
0620 07:10:07.162682 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":390,"threads_started":1}
I20260620 07:10:07.163369 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.126218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.67:48815)
0620 07:10:07.127224 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:07.127247 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:07.128318 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:07.132156 (+ 3838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:07.132174 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:07.133314 (+ 1140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:07.134480 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:07.134513 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.138948 (+ 4435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.138955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:07.141479 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:07.141495 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.142760 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.142767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:07.143256 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.144383 (+ 1127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:07.144428 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:07.147914 (+ 3486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:07.153660 (+ 5746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:07.153669 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:07.153689 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:07.154324 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:07.154746 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:07.154751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:07.154755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:07.155000 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:07.155748 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:07.155757 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:07.156185 (+ 428us) client_negotiation.cc:769] Sending connection context
0620 07:10:07.156492 (+ 307us) client_negotiation.cc:240] Negotiation successful
0620 07:10:07.160546 (+ 4054us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":131,"thread_start_us":553,"threads_started":1}
I20260620 07:10:07.165297 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.123760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:44185 (local address 127.6.53.124:36739)
0620 07:10:07.124444 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:07.124450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:07.125319 (+ 869us) server_negotiation.cc:407] Connection header received
0620 07:10:07.138315 (+ 12996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:07.138324 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:07.138438 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:07.138517 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:07.140815 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.142231 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.145163 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.146225 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.146364 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.156025 (+ 9661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:07.156035 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:07.156043 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:07.156101 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:07.160363 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.161919 (+ 1556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.161926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.161929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.162122 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.162880 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.162888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.162894 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.163382 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:07.163636 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:07.164302 (+ 666us) server_negotiation.cc:299] Negotiation successful
0620 07:10:07.164706 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":406,"threads_started":1}
I20260620 07:10:07.165779 13771 heartbeater.cc:347] Connected to a master server at 127.6.53.126:41085
I20260620 07:10:07.166320 13771 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:07.166996 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.123930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36739 (local address 127.6.53.67:44185)
0620 07:10:07.125162 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:07.125208 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:07.138079 (+ 12871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:07.138737 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:07.138743 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:07.139818 (+ 1075us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:07.140597 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:07.140615 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.142495 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.142503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:07.144807 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:07.144848 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.146682 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.146689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:07.146884 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.147690 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:07.147713 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:07.155701 (+ 7988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:07.160690 (+ 4989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:07.160700 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:07.160706 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:07.161734 (+ 1028us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:07.162412 (+ 678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:07.162419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:07.162436 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:07.162726 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:07.163752 (+ 1026us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:07.163756 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:07.163916 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 07:10:07.165730 (+ 1814us) client_negotiation.cc:240] Negotiation successful
0620 07:10:07.166168 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":6,"thread_start_us":644,"threads_started":1}
I20260620 07:10:07.167625 13771 heartbeater.cc:511] Master 127.6.53.126:41085 requested a full tablet report, sending...
I20260620 07:10:07.169234 13770 heartbeater.cc:347] Connected to a master server at 127.6.53.124:36739
I20260620 07:10:07.169389 13770 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:07.169844 13770 heartbeater.cc:511] Master 127.6.53.124:36739 requested a full tablet report, sending...
I20260620 07:10:07.171068 13303 ts_manager.cc:194] Registered new tserver with Master: ca2ec70afce24c1099d2f07d015010ec (127.6.53.67:42697)
I20260620 07:10:07.172772 12511 ts_manager.cc:194] Re-registered known tserver with Master: ca2ec70afce24c1099d2f07d015010ec (127.6.53.67:42697)
I20260620 07:10:07.174325 12511 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:48815
Jun 20 07:10:07 dist-test-slave-4n6h krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939407, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.125@KRBTEST.COM
I20260620 07:10:07.186355 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.129483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39562 (local address 127.6.53.126:41085)
0620 07:10:07.137619 (+ 8136us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:07.137624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:07.138965 (+ 1341us) server_negotiation.cc:407] Connection header received
0620 07:10:07.139018 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:07.139024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:07.139116 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:07.139217 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:07.143306 (+ 4089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.144478 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.147562 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.148228 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.148353 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.159272 (+ 10919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:07.159283 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:07.159289 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:07.159335 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:07.175128 (+ 15793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.182352 (+ 7224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.182359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.182364 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.182517 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.183752 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.183757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.183759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.184146 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:07.184425 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:07.185602 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0620 07:10:07.185887 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8006,"thread_start_us":7878,"threads_started":1}
I20260620 07:10:07.187947 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.138882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56442 (local address 127.6.53.125:38109)
0620 07:10:07.146281 (+ 7399us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:07.146287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:07.146852 (+ 565us) server_negotiation.cc:407] Connection header received
0620 07:10:07.164516 (+ 17664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:07.164525 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:07.164620 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:07.164700 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:07.168036 (+ 3336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.169255 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.171252 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.171911 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.172092 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.177755 (+ 5663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:07.177762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:07.177767 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:07.177817 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:07.181279 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.182338 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.182345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.182349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.182554 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.185579 (+ 3025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.185585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.185590 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.186052 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:07.186561 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:07.187306 (+ 745us) server_negotiation.cc:299] Negotiation successful
0620 07:10:07.187596 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5075,"thread_start_us":5405,"threads_started":1}
I20260620 07:10:07.195026 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.147736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.6.53.124:36739)
0620 07:10:07.165671 (+ 17935us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:07.165679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:07.174801 (+ 9122us) server_negotiation.cc:407] Connection header received
0620 07:10:07.175821 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:07.175828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:07.175929 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:07.176004 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:07.178326 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.179216 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.180497 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.180897 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.181031 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.185865 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:07.185872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:07.185877 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:07.185922 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:07.190421 (+ 4499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.191362 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.191371 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.191375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.191672 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.192504 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.192510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.192513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.193123 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:07.193441 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:07.194152 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 07:10:07.194478 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17807,"thread_start_us":25312,"threads_started":1}
I20260620 07:10:07.195999 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.127964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:38109 (local address 127.6.53.67:40765)
0620 07:10:07.138040 (+ 10076us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:07.138057 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:07.138175 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:07.141986 (+ 3811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:07.141992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:07.142589 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:07.143114 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:07.143127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.148545 (+ 5418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.148551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:07.150855 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:07.150871 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.152200 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.152208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:07.152402 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.180898 (+ 28496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:07.180923 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:07.183534 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:07.192382 (+ 8848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:07.192390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:07.192397 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:07.193095 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:07.193546 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:07.193550 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:07.193553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:07.193691 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:07.194430 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:07.194437 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:07.194732 (+ 295us) client_negotiation.cc:769] Sending connection context
0620 07:10:07.194952 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 07:10:07.195565 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9929,"thread_start_us":9828,"threads_started":1}
I20260620 07:10:07.199530 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.127342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:40765 (local address 127.6.53.125:38109)
0620 07:10:07.137534 (+ 10192us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:07.137543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:07.141552 (+ 4009us) server_negotiation.cc:407] Connection header received
0620 07:10:07.141620 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:07.141625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:07.141713 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:07.141805 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:07.146333 (+ 4528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.147391 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.151037 (+ 3646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.151822 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.151958 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.183821 (+ 31863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:07.183829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:07.183836 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:07.183916 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:07.192128 (+ 8212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.193250 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.193256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.193260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.193404 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.193815 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.193820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.193824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.194188 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:07.194415 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:07.198860 (+ 4445us) server_negotiation.cc:299] Negotiation successful
0620 07:10:07.199115 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9679,"thread_start_us":9781,"threads_started":1}
I20260620 07:10:07.200547 13772 heartbeater.cc:347] Connected to a master server at 127.6.53.125:38109
I20260620 07:10:07.200743 13772 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:07.201152 13772 heartbeater.cc:511] Master 127.6.53.125:38109 requested a full tablet report, sending...
I20260620 07:10:07.202982 13109 ts_manager.cc:194] Registered new tserver with Master: ca2ec70afce24c1099d2f07d015010ec (127.6.53.67:42697)
I20260620 07:10:07.195999 12511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:10:07.207182 12511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:10:07.242684 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.229869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43768 (local address 127.6.53.67:42697)
0620 07:10:07.230747 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:07.230755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:07.231830 (+ 1075us) server_negotiation.cc:407] Connection header received
0620 07:10:07.232111 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:07.232120 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:07.232386 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:07.232619 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:07.233673 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.236103 (+ 2430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.238925 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.240789 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.240923 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.241141 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:07.242076 (+ 935us) server_negotiation.cc:299] Negotiation successful
0620 07:10:07.242290 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":904,"threads_started":1}
I20260620 07:10:07.242950 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.229357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:42697 (local address 127.0.0.1:43768)
0620 07:10:07.231698 (+ 2341us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:07.231715 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:07.231896 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:07.232595 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:07.232599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:07.232627 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:07.233128 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:07.233143 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.236348 (+ 3205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.236354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:07.238726 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:07.238741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.241024 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.241033 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:07.241235 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.241579 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 07:10:07.242001 (+ 422us) client_negotiation.cc:240] Negotiation successful
0620 07:10:07.242158 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2185,"thread_start_us":10531,"threads_started":1}
I20260620 07:10:07.249573 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.227986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:39055 (local address 127.0.0.1:56700)
0620 07:10:07.229018 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:07.229047 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:07.236595 (+ 7548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:07.238909 (+ 2314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:07.238914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:07.238944 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:07.239314 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:07.239324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.241320 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.241323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:07.245595 (+ 4272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:07.245612 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.248080 (+ 2468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.248088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:07.248563 (+ 475us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.248611 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 07:10:07.248905 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 07:10:07.249148 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":256,"thread_start_us":712,"threads_started":1}
I20260620 07:10:07.248960 13705 tablet_service.cc:1467] Processing CreateTablet for tablet 01da1a346be4460987e843aa1c2a508e (DEFAULT_TABLE table=test-table [id=995ca2270ed04cd59f73f012477164a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:07.249827 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.228332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.6.53.66:39055)
0620 07:10:07.230099 (+ 1767us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:07.230111 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:07.230156 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:10:07.236875 (+ 6719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:07.236887 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:07.237214 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:07.237496 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:07.239567 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.241125 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.246057 (+ 4932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.247751 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.247993 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.248268 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:07.249065 (+ 797us) server_negotiation.cc:299] Negotiation successful
0620 07:10:07.249304 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1593,"thread_start_us":1434,"threads_started":1}
I20260620 07:10:07.250902 13705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01da1a346be4460987e843aa1c2a508e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:07.254321 13564 tablet_service.cc:1467] Processing CreateTablet for tablet 01da1a346be4460987e843aa1c2a508e (DEFAULT_TABLE table=test-table [id=995ca2270ed04cd59f73f012477164a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:07.255959 13564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01da1a346be4460987e843aa1c2a508e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:07.257642 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.230556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47726 (local address 127.6.53.65:42627)
0620 07:10:07.242033 (+ 11477us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:07.242043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:07.242078 (+ 35us) server_negotiation.cc:407] Connection header received
0620 07:10:07.242183 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:07.242191 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:07.242559 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:07.242775 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:07.243827 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.246088 (+ 2261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.254123 (+ 8035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.256424 (+ 2301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.256629 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.256869 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:07.257134 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 07:10:07.257315 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10649,"thread_start_us":12367,"threads_started":1}
I20260620 07:10:07.257678 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.229355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42627 (local address 127.0.0.1:47726)
0620 07:10:07.240209 (+ 10854us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:07.240224 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:07.240323 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:07.243202 (+ 2879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:07.243211 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:07.243241 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:07.243655 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:07.243664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.246404 (+ 2740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.246410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:07.250871 (+ 4461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:07.250901 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.256686 (+ 5785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.256694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:07.256922 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.256959 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 07:10:07.257080 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 07:10:07.257225 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10714,"mutex_wait_us":344,"thread_start_us":1732,"threads_started":1}
I20260620 07:10:07.261081 13415 tablet_service.cc:1467] Processing CreateTablet for tablet 01da1a346be4460987e843aa1c2a508e (DEFAULT_TABLE table=test-table [id=995ca2270ed04cd59f73f012477164a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:07.262843 13415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01da1a346be4460987e843aa1c2a508e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:07.268766 13806 tablet_bootstrap.cc:492] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec: Bootstrap starting.
I20260620 07:10:07.273001 13806 tablet_bootstrap.cc:654] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:07.274307 13806 log.cc:824] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:07.274843 13807 tablet_bootstrap.cc:492] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc: Bootstrap starting.
I20260620 07:10:07.277700 13809 tablet_bootstrap.cc:492] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc: Bootstrap starting.
I20260620 07:10:07.278677 13807 tablet_bootstrap.cc:654] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:07.280129 13807 log.cc:824] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:07.280925 13806 tablet_bootstrap.cc:492] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec: No bootstrap required, opened a new log
I20260620 07:10:07.281401 13806 ts_tablet_manager.cc:1397] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260620 07:10:07.282992 13809 tablet_bootstrap.cc:654] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:07.284240 13807 tablet_bootstrap.cc:492] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc: No bootstrap required, opened a new log
I20260620 07:10:07.284536 13807 ts_tablet_manager.cc:1397] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 07:10:07.285310 13809 log.cc:824] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:07.288839 13809 tablet_bootstrap.cc:492] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc: No bootstrap required, opened a new log
I20260620 07:10:07.289150 13809 ts_tablet_manager.cc:1397] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260620 07:10:07.293303 13806 raft_consensus.cc:348] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:07.293749 13806 raft_consensus.cc:374] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:07.293879 13806 raft_consensus.cc:729] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2ec70afce24c1099d2f07d015010ec, State: Initialized, Role: FOLLOWER
I20260620 07:10:07.294761 13806 consensus_queue.cc:260] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:07.295696 13807 raft_consensus.cc:348] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:07.296243 13807 raft_consensus.cc:374] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:07.296358 13807 raft_consensus.cc:729] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42141b2ca9ed4593a530955665d7a0fc, State: Initialized, Role: FOLLOWER
I20260620 07:10:07.297045 13807 consensus_queue.cc:260] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:07.297358 13809 raft_consensus.cc:348] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:07.297783 13809 raft_consensus.cc:374] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:07.297884 13809 raft_consensus.cc:729] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659e9bb195db4166b7c0c6bc30452fbc, State: Initialized, Role: FOLLOWER
I20260620 07:10:07.298578 13809 consensus_queue.cc:260] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:07.298817 13806 ts_tablet_manager.cc:1428] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec: Time spent starting tablet: real 0.017s user 0.019s sys 0.000s
I20260620 07:10:07.299607 13771 heartbeater.cc:503] Master 127.6.53.126:41085 was elected leader, sending a full tablet report...
I20260620 07:10:07.303573 13809 ts_tablet_manager.cc:1428] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 07:10:07.307346 13630 heartbeater.cc:503] Master 127.6.53.126:41085 was elected leader, sending a full tablet report...
I20260620 07:10:07.310034 13807 ts_tablet_manager.cc:1428] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc: Time spent starting tablet: real 0.025s user 0.024s sys 0.001s
W20260620 07:10:07.312357 13491 tablet.cc:2369] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:10:07.347795 13813 raft_consensus.cc:482] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:10:07.348147 13813 raft_consensus.cc:504] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:07.350054 13813 leader_election.cc:288] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 659e9bb195db4166b7c0c6bc30452fbc (127.6.53.65:42627), ca2ec70afce24c1099d2f07d015010ec (127.6.53.67:42697)
I20260620 07:10:07.359381 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.350684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:46371 (local address 127.6.53.67:42697)
0620 07:10:07.350915 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:07.350919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:07.351530 (+ 611us) server_negotiation.cc:407] Connection header received
0620 07:10:07.351846 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:07.351853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:07.351940 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:07.352029 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:07.352720 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.353812 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.356794 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.358097 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.358234 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.358477 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:07.358985 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 07:10:07.359123 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 07:10:07.359380 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.350610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:60311 (local address 127.6.53.65:42627)
0620 07:10:07.350829 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:07.350838 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:07.352091 (+ 1253us) server_negotiation.cc:407] Connection header received
0620 07:10:07.352273 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:07.352277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:07.352334 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:07.352720 (+ 386us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:07.353193 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.354135 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.356831 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.358015 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.358151 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.358445 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:07.358984 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 07:10:07.359122 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:10:07.359683 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.350596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42627 (local address 127.6.53.66:60311)
0620 07:10:07.351961 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:07.351975 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:07.352116 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:07.352766 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:07.352769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:07.352791 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:07.353038 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:07.353046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.354289 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.354297 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:07.356587 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:07.356596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.358582 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.358587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:07.358733 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.358823 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 07:10:07.358940 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 07:10:07.359106 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1270,"mutex_wait_us":58,"spinlock_wait_cycles":14592,"thread_start_us":1081,"threads_started":1}
I20260620 07:10:07.359709 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.350597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:42697 (local address 127.6.53.66:46371)
0620 07:10:07.351405 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:07.351422 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:07.351633 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:07.352086 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:07.352091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:07.352135 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:07.352546 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:07.352557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.354001 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.354005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:07.356574 (+ 2569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:07.356592 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.358501 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.358510 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:07.358703 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.358825 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 07:10:07.358985 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 07:10:07.359116 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":83,"spinlock_wait_cycles":3328,"thread_start_us":454,"threads_started":1}
I20260620 07:10:07.360711 13435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01da1a346be4460987e843aa1c2a508e" candidate_uuid: "42141b2ca9ed4593a530955665d7a0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659e9bb195db4166b7c0c6bc30452fbc" is_pre_election: true
I20260620 07:10:07.360773 13725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01da1a346be4460987e843aa1c2a508e" candidate_uuid: "42141b2ca9ed4593a530955665d7a0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca2ec70afce24c1099d2f07d015010ec" is_pre_election: true
I20260620 07:10:07.361234 13435 raft_consensus.cc:2454] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42141b2ca9ed4593a530955665d7a0fc in term 0.
I20260620 07:10:07.361331 13725 raft_consensus.cc:2454] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42141b2ca9ed4593a530955665d7a0fc in term 0.
I20260620 07:10:07.362008 13517 leader_election.cc:302] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 42141b2ca9ed4593a530955665d7a0fc, 659e9bb195db4166b7c0c6bc30452fbc, ca2ec70afce24c1099d2f07d015010ec; no voters:
I20260620 07:10:07.362557 13813 raft_consensus.cc:2781] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:10:07.362771 13813 raft_consensus.cc:482] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:10:07.362893 13813 raft_consensus.cc:3037] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:07.365063 13813 raft_consensus.cc:504] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:07.365631 13813 leader_election.cc:288] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [CANDIDATE]: Term 1 election: Requested vote from peers 659e9bb195db4166b7c0c6bc30452fbc (127.6.53.65:42627), ca2ec70afce24c1099d2f07d015010ec (127.6.53.67:42697)
I20260620 07:10:07.366492 13725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01da1a346be4460987e843aa1c2a508e" candidate_uuid: "42141b2ca9ed4593a530955665d7a0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca2ec70afce24c1099d2f07d015010ec"
I20260620 07:10:07.366585 13435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01da1a346be4460987e843aa1c2a508e" candidate_uuid: "42141b2ca9ed4593a530955665d7a0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659e9bb195db4166b7c0c6bc30452fbc"
I20260620 07:10:07.366719 13725 raft_consensus.cc:3037] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:07.366770 13435 raft_consensus.cc:3037] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:07.370045 13435 raft_consensus.cc:2454] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42141b2ca9ed4593a530955665d7a0fc in term 1.
I20260620 07:10:07.370046 13725 raft_consensus.cc:2454] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42141b2ca9ed4593a530955665d7a0fc in term 1.
I20260620 07:10:07.370513 13518 leader_election.cc:302] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42141b2ca9ed4593a530955665d7a0fc, ca2ec70afce24c1099d2f07d015010ec; no voters:
I20260620 07:10:07.370884 13813 raft_consensus.cc:2781] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:10:07.371356 13813 raft_consensus.cc:686] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [term 1 LEADER]: Becoming Leader. State: Replica: 42141b2ca9ed4593a530955665d7a0fc, State: Running, Role: LEADER
I20260620 07:10:07.372195 13813 consensus_queue.cc:237] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
W20260620 07:10:07.373708 13773 tablet.cc:2369] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:10:07.377707 12511 catalog_manager.cc:5495] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 42141b2ca9ed4593a530955665d7a0fc (127.6.53.66). New cstate: current_term: 1 leader_uuid: "42141b2ca9ed4593a530955665d7a0fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } health_report { overall_health: UNKNOWN } } }
W20260620 07:10:07.396126 13632 tablet.cc:2369] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:10:07.403421 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.396358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.6.53.66:39055)
0620 07:10:07.396993 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:07.397001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:07.397020 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:10:07.397116 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:07.397119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:07.397190 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:07.397348 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:10:07.397726 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.398645 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.400552 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.401118 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.401293 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.401893 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:10:07.402709 (+ 816us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:10:07.402912 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:07.402978 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 07:10:07.403078 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498}
I20260620 07:10:07.412134 13725 raft_consensus.cc:1261] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec [term 1 FOLLOWER]: Refusing update from remote peer 42141b2ca9ed4593a530955665d7a0fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:10:07.412518 13435 raft_consensus.cc:1261] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc [term 1 FOLLOWER]: Refusing update from remote peer 42141b2ca9ed4593a530955665d7a0fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:10:07.413157 13813 consensus_queue.cc:1035] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:07.413690 13817 consensus_queue.cc:1035] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:07.431916 13822 mvcc.cc:205] Tried to move back new op lower bound from 7298823812749762560 to 7298823812600721408. Current Snapshot: MvccSnapshot[applied={T|T < 7298823812749762560}]
I20260620 07:10:07.447507 13823 mvcc.cc:205] Tried to move back new op lower bound from 7298823812749762560 to 7298823812600721408. Current Snapshot: MvccSnapshot[applied={T|T < 7298823812749762560}]
I20260620 07:10:07.459692 12511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:10:07.460021 12511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:10:07.471427 12511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f0bb53d5db24273b564b44d83a922bc: Sending DeleteTablet for 3 replicas of tablet 01da1a346be4460987e843aa1c2a508e
I20260620 07:10:07.473147 13564 tablet_service.cc:1514] Processing DeleteTablet for tablet 01da1a346be4460987e843aa1c2a508e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:07 UTC) from {username='oryx', principal='oryx/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:56700
I20260620 07:10:07.473610 13705 tablet_service.cc:1514] Processing DeleteTablet for tablet 01da1a346be4460987e843aa1c2a508e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:07 UTC) from {username='oryx', principal='oryx/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:43768
I20260620 07:10:07.473886 13415 tablet_service.cc:1514] Processing DeleteTablet for tablet 01da1a346be4460987e843aa1c2a508e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:07 UTC) from {username='oryx', principal='oryx/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:47726
I20260620 07:10:07.487075 13831 tablet_replica.cc:330] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec: stopping tablet replica
I20260620 07:10:07.488401 13831 raft_consensus.cc:2229] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:10:07.489370 13831 raft_consensus.cc:2258] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:07.490809 13830 tablet_replica.cc:330] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc: stopping tablet replica
I20260620 07:10:07.491331 13830 raft_consensus.cc:2229] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:10:07.492225 13830 raft_consensus.cc:2258] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:07.495294 13830 ts_tablet_manager.cc:1905] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:07.495290 13831 ts_tablet_manager.cc:1905] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:07.504171 13831 ts_tablet_manager.cc:1918] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:10:07.504388 13831 log.cc:1196] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/01da1a346be4460987e843aa1c2a508e
I20260620 07:10:07.504954 13831 ts_tablet_manager.cc:1939] T 01da1a346be4460987e843aa1c2a508e P ca2ec70afce24c1099d2f07d015010ec: Deleting consensus metadata
I20260620 07:10:07.505638 13830 ts_tablet_manager.cc:1918] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:10:07.505906 13830 log.cc:1196] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/01da1a346be4460987e843aa1c2a508e
I20260620 07:10:07.506021 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 12479
I20260620 07:10:07.507023 13830 ts_tablet_manager.cc:1939] T 01da1a346be4460987e843aa1c2a508e P 42141b2ca9ed4593a530955665d7a0fc: Deleting consensus metadata
I20260620 07:10:07.507365 12541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:07.507687 13832 tablet_replica.cc:330] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc: stopping tablet replica
I20260620 07:10:07.508361 13832 raft_consensus.cc:2229] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:10:07.509083 13832 raft_consensus.cc:2258] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:07.511976 13832 ts_tablet_manager.cc:1905] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:07.519685 13832 ts_tablet_manager.cc:1918] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:10:07.520190 13832 log.cc:1196] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/01da1a346be4460987e843aa1c2a508e
I20260620 07:10:07.520911 13832 ts_tablet_manager.cc:1939] T 01da1a346be4460987e843aa1c2a508e P 659e9bb195db4166b7c0c6bc30452fbc: Deleting consensus metadata
I20260620 07:10:07.527093 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.477019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.6.53.125:38109)
0620 07:10:07.477207 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:07.477213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:07.501731 (+ 24518us) server_negotiation.cc:407] Connection header received
0620 07:10:07.502801 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:07.502809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:07.502911 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:07.502988 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:07.510429 (+ 7441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.511868 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.514301 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.515133 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.515243 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.519076 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:07.519083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:07.519088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:07.519133 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:07.523420 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.524623 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.524630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.524635 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.524802 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.525362 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.525367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.525370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.525716 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:07.525945 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:07.526480 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 07:10:07.526745 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:10:07.546550 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.502367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45516 (local address 127.6.53.124:36739)
0620 07:10:07.502558 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:07.502566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:07.527332 (+ 24766us) server_negotiation.cc:407] Connection header received
0620 07:10:07.527973 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:07.527980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:07.528074 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:07.528153 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:07.530686 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.531977 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.534189 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.534999 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.535145 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.538519 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:07.538526 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:07.538531 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:07.538572 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:07.542551 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.543407 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.543413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.543417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.543572 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.544067 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.544072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.544076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.544483 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:07.544978 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:07.545266 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 07:10:07.545924 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:10:07.659036 12495 catalog_manager.cc:4841] TS 42141b2ca9ed4593a530955665d7a0fc (127.6.53.66:39055): tablet 01da1a346be4460987e843aa1c2a508e (table test-table [id=995ca2270ed04cd59f73f012477164a3]) successfully deleted
W20260620 07:10:07.659373 12495 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.53.66:39055, user_credentials={real_user=slave}} blocked reactor thread for 150171us
I20260620 07:10:07.659795 12495 catalog_manager.cc:4841] TS 659e9bb195db4166b7c0c6bc30452fbc (127.6.53.65:42627): tablet 01da1a346be4460987e843aa1c2a508e (table test-table [id=995ca2270ed04cd59f73f012477164a3]) successfully deleted
I20260620 07:10:07.659838 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 12479
I20260620 07:10:07.711977 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.693072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.6.53.125:38109)
0620 07:10:07.693323 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:07.693333 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:07.693453 (+ 120us) server_negotiation.cc:407] Connection header received
0620 07:10:07.693762 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:07.693771 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:07.693876 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:07.694024 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:07.696706 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.697916 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.699272 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.699891 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.700039 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.704238 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:07.704246 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:07.704250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:07.704294 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:07.708181 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.709364 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.709371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.709375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.709657 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.710210 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.710216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.710220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.710671 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:07.711079 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:07.711230 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 07:10:07.711511 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:10:07.715224 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:07.694792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45528 (local address 127.6.53.124:36739)
0620 07:10:07.694991 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:07.694996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:07.695281 (+ 285us) server_negotiation.cc:407] Connection header received
0620 07:10:07.695492 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:07.695499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:07.695566 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:07.695676 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:07.698016 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.699299 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.701658 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:07.702140 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:07.702436 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:07.705408 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:07.705419 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:07.705423 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:07.705466 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:07.709162 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.709805 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.709813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.709818 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.709981 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:07.712678 (+ 2697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:07.712685 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:07.712688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:07.713125 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:07.713462 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:07.714176 (+ 714us) server_negotiation.cc:299] Negotiation successful
0620 07:10:07.714570 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:10:08.452520 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:08.451693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.66:57497)
0620 07:10:08.452173 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:08.452320 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":198,"threads_started":1}
W20260620 07:10:08.452937 13630 heartbeater.cc:644] Failed to heartbeat to 127.6.53.126:41085 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
I20260620 07:10:08.454530 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:08.453776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.67:38245)
0620 07:10:08.454200 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:08.454372 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":247,"threads_started":1}
W20260620 07:10:08.454927 13771 heartbeater.cc:644] Failed to heartbeat to 127.6.53.126:41085 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
I20260620 07:10:08.457022 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:08.456424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.65:39759)
0620 07:10:08.456785 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:08.456887 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":206,"threads_started":1}
W20260620 07:10:08.457326 13487 heartbeater.cc:644] Failed to heartbeat to 127.6.53.126:41085 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
I20260620 07:10:08.970350 13844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f0bb53d5db24273b564b44d83a922bc)
I20260620 07:10:08.970594 13844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } }
I20260620 07:10:08.970862 13845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f0bb53d5db24273b564b44d83a922bc)
I20260620 07:10:08.971127 13845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } }
I20260620 07:10:08.971622 13844 leader_election.cc:288] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f0bb53d5db24273b564b44d83a922bc (127.6.53.126:41085), 6fc80693e6e44dc9a0e14b8bc5eb4b01 (127.6.53.124:36739)
I20260620 07:10:08.972093 13845 leader_election.cc:288] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f0bb53d5db24273b564b44d83a922bc (127.6.53.126:41085), 0963332cf16b489e9eaf2607296f0e60 (127.6.53.125:38109)
I20260620 07:10:08.973027 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:08.971753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.0.0.1:39966)
0620 07:10:08.972543 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:08.972778 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":528,"threads_started":1}
I20260620 07:10:08.973276 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:08.972199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.0.0.1:39974)
0620 07:10:08.972784 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:08.972948 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":333,"threads_started":1}
W20260620 07:10:08.976912 13095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
W20260620 07:10:08.979774 13289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
I20260620 07:10:08.984073 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:08.972723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36739 (local address 127.0.0.1:45544)
0620 07:10:08.973224 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:08.973241 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:08.973381 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:08.974948 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:08.974953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:08.974979 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:08.975210 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:08.975229 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:08.977234 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:08.977242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:08.980277 (+ 3035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:08.980296 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:08.982880 (+ 2584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:08.982889 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:08.983144 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:08.983198 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 07:10:08.983344 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 07:10:08.983550 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":271,"thread_start_us":2011,"threads_started":1}
I20260620 07:10:08.984989 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:08.973161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.6.53.124:36739)
0620 07:10:08.974189 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:08.974197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:08.974218 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:10:08.974315 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:08.974322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:08.974384 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:08.974474 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:08.975461 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:08.977039 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:08.980505 (+ 3466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:08.982452 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:08.982604 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:08.982748 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:08.984283 (+ 1535us) server_negotiation.cc:299] Negotiation successful
0620 07:10:08.984483 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":888,"thread_start_us":1159,"threads_started":1}
I20260620 07:10:08.985965 13313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0963332cf16b489e9eaf2607296f0e60" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" is_pre_election: true
I20260620 07:10:08.986284 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:08.972918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:38109 (local address 127.0.0.1:56466)
0620 07:10:08.973492 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:08.973512 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:08.973665 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:08.977225 (+ 3560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:08.977230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:08.977273 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:08.977672 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:08.977687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:08.980068 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:08.980077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:08.983047 (+ 2970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:08.983066 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:08.985406 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:08.985412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:08.985629 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:08.985663 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 07:10:08.985747 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 07:10:08.985895 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":6,"thread_start_us":6574,"threads_started":1}
I20260620 07:10:08.986372 13313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0963332cf16b489e9eaf2607296f0e60 in term 1.
I20260620 07:10:08.986792 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:08.974953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.6.53.125:38109)
0620 07:10:08.976926 (+ 1973us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:08.976933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:08.976952 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:10:08.977006 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:08.977012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:08.977078 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:08.977853 (+ 775us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:08.978160 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:08.979857 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:08.983293 (+ 3436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:08.985193 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:08.986011 (+ 818us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:08.986154 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:08.986203 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 07:10:08.986391 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1855,"thread_start_us":1706,"threads_started":1}
I20260620 07:10:08.987213 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:08.986846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.0.0.1:39990)
0620 07:10:08.987006 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:08.987098 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 07:10:08.987531 13095 leader_election.cc:302] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0963332cf16b489e9eaf2607296f0e60, 6fc80693e6e44dc9a0e14b8bc5eb4b01; no voters:
I20260620 07:10:08.987836 13844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260620 07:10:08.987890 13095 leader_election.cc:334] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8f0bb53d5db24273b564b44d83a922bc (127.6.53.126:41085): Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
I20260620 07:10:08.987959 13844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f0bb53d5db24273b564b44d83a922bc)
I20260620 07:10:08.988072 13844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:10:08.988157 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:08.987812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.0.0.1:40002)
0620 07:10:08.987940 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:08.988037 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 07:10:08.988319 13289 leader_election.cc:334] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8f0bb53d5db24273b564b44d83a922bc (127.6.53.126:41085): Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
I20260620 07:10:08.988535 13118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0963332cf16b489e9eaf2607296f0e60" is_pre_election: true
I20260620 07:10:08.990399 13844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } }
I20260620 07:10:08.990762 13118 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6fc80693e6e44dc9a0e14b8bc5eb4b01 in current term 2: Already voted for candidate 0963332cf16b489e9eaf2607296f0e60 in this term.
I20260620 07:10:08.991039 13844 leader_election.cc:288] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [CANDIDATE]: Term 2 election: Requested vote from peers 8f0bb53d5db24273b564b44d83a922bc (127.6.53.126:41085), 6fc80693e6e44dc9a0e14b8bc5eb4b01 (127.6.53.124:36739)
I20260620 07:10:08.991492 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:08.991081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.0.0.1:40006)
0620 07:10:08.991251 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:08.991332 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 07:10:08.991631 13313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0963332cf16b489e9eaf2607296f0e60" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01"
I20260620 07:10:08.991714 13289 leader_election.cc:302] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fc80693e6e44dc9a0e14b8bc5eb4b01; no voters: 0963332cf16b489e9eaf2607296f0e60, 8f0bb53d5db24273b564b44d83a922bc
I20260620 07:10:08.991885 13313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:10:08.992427 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:08.992118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.0.0.1:40020)
0620 07:10:08.992262 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:08.992322 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 07:10:08.992575 13095 leader_election.cc:334] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8f0bb53d5db24273b564b44d83a922bc (127.6.53.126:41085): Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
I20260620 07:10:08.994047 13313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0963332cf16b489e9eaf2607296f0e60 in term 2.
I20260620 07:10:08.994298 13845 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:10:08.994509 13095 leader_election.cc:302] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0963332cf16b489e9eaf2607296f0e60, 6fc80693e6e44dc9a0e14b8bc5eb4b01; no voters: 8f0bb53d5db24273b564b44d83a922bc
I20260620 07:10:08.994793 13844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:10:08.995194 13844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [term 2 LEADER]: Becoming Leader. State: Replica: 0963332cf16b489e9eaf2607296f0e60, State: Running, Role: LEADER
I20260620 07:10:08.995551 13844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } }
I20260620 07:10:08.997406 13854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0963332cf16b489e9eaf2607296f0e60. Latest consensus state: current_term: 2 leader_uuid: "0963332cf16b489e9eaf2607296f0e60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } } }
I20260620 07:10:08.997795 13854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:08.998549 13856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:10:09.001919 13856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:10:09.002786 13856 catalog_manager.cc:1259] Loaded cluster ID: 164f487c8df14824bf1945fe4a35bc1b
I20260620 07:10:09.002856 13856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:10:09.004014 13856 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:10:09.004812 13856 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: Loaded TSK: 0
I20260620 07:10:09.005673 13856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:10:09.035796 13109 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:10:09.042927 13109 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:10:09.046926 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.046516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.0.0.1:40024)
0620 07:10:09.046676 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.046765 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 07:10:09.046911 13313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [term 2 FOLLOWER]: Refusing update from remote peer 0963332cf16b489e9eaf2607296f0e60: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 07:10:09.048031 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.047601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.0.0.1:40030)
0620 07:10:09.047734 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.047864 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 07:10:09.047928 13854 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 07:10:09.048198 13095 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 -> Peer 8f0bb53d5db24273b564b44d83a922bc (127.6.53.126:41085): Couldn't send request to peer 8f0bb53d5db24273b564b44d83a922bc. Status: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 07:10:09.050280 13845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0963332cf16b489e9eaf2607296f0e60. Latest consensus state: current_term: 2 leader_uuid: "0963332cf16b489e9eaf2607296f0e60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } } }
I20260620 07:10:09.050581 13845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:10:09.052145 13844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "0963332cf16b489e9eaf2607296f0e60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } } }
I20260620 07:10:09.052450 13844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60 [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:09.052975 13845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "0963332cf16b489e9eaf2607296f0e60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f0bb53d5db24273b564b44d83a922bc" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41085 } } peers { permanent_uuid: "0963332cf16b489e9eaf2607296f0e60" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 38109 } attrs { promote: false } } peers { permanent_uuid: "6fc80693e6e44dc9a0e14b8bc5eb4b01" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36739 } attrs { promote: false } } }
I20260620 07:10:09.053200 13845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc80693e6e44dc9a0e14b8bc5eb4b01 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:10:09.070494 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.061821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.6.53.67:42697)
0620 07:10:09.064088 (+ 2267us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:09.064097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:09.064116 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:10:09.064182 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:09.064188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:09.064272 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:09.064389 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:09.065025 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.066073 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.068210 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.069370 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.069491 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:09.069596 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:09.070021 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 07:10:09.070148 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2122,"thread_start_us":2013,"threads_started":1}
I20260620 07:10:09.070580 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.061571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:42697 (local address 127.0.0.1:43778)
0620 07:10:09.063457 (+ 1886us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.063478 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:09.063607 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:09.064410 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:09.064415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:09.064438 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:09.064860 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:09.064877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.066219 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.066224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:09.068064 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:09.068095 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.069722 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.069726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:09.069857 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:09.069880 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:10:09.069987 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 07:10:09.070110 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1736,"mutex_wait_us":258,"thread_start_us":1263,"threads_started":1}
I20260620 07:10:09.071100 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.061554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:39055 (local address 127.0.0.1:56712)
0620 07:10:09.061851 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.061871 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:09.061967 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:09.064616 (+ 2649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:09.064620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:09.064640 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:09.065195 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:09.065201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.066481 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.066488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:09.068710 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:09.068724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.070350 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.070356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:09.070555 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:09.070589 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 07:10:09.070662 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 07:10:09.070814 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":230}
I20260620 07:10:09.071390 13705 tablet_service.cc:1467] Processing CreateTablet for tablet 70b03b5f7c444bc8aeca389992d60c4e (DEFAULT_TABLE table=test-table [id=4f9ee5d3482047659808a1d246897eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:09.071823 13705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b03b5f7c444bc8aeca389992d60c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:09.071928 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.060389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42627 (local address 127.0.0.1:47736)
0620 07:10:09.060716 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.060739 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:09.060889 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:09.064647 (+ 3758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:09.064651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:09.064679 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:09.064895 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:09.064904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.066221 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.066226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:09.068425 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:09.068438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.070242 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.070250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:09.070491 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:09.071433 (+ 942us) client_negotiation.cc:769] Sending connection context
0620 07:10:09.071509 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 07:10:09.071680 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":931}
I20260620 07:10:09.072059 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.061915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.6.53.66:39055)
0620 07:10:09.062654 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:09.062663 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:09.062686 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:10:09.062771 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:09.062779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:09.062873 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:09.064795 (+ 1922us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:09.065325 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.066348 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.068836 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.070029 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.071440 (+ 1411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:09.071568 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:09.071617 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 07:10:09.071743 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":509,"threads_started":1}
I20260620 07:10:09.072718 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.060512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47736 (local address 127.6.53.65:42627)
0620 07:10:09.064310 (+ 3798us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:09.064316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:09.064334 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:10:09.064402 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:09.064407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:09.064490 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:09.065323 (+ 833us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:09.065430 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.066086 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.068880 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.070041 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.072238 (+ 2197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:09.072333 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:09.072373 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 07:10:09.072527 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3676,"thread_start_us":3574,"threads_started":1}
I20260620 07:10:09.072731 13564 tablet_service.cc:1467] Processing CreateTablet for tablet 70b03b5f7c444bc8aeca389992d60c4e (DEFAULT_TABLE table=test-table [id=4f9ee5d3482047659808a1d246897eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:09.073135 13564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b03b5f7c444bc8aeca389992d60c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:09.078420 13867 tablet_bootstrap.cc:492] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec: Bootstrap starting.
I20260620 07:10:09.079449 13415 tablet_service.cc:1467] Processing CreateTablet for tablet 70b03b5f7c444bc8aeca389992d60c4e (DEFAULT_TABLE table=test-table [id=4f9ee5d3482047659808a1d246897eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:09.079638 13868 tablet_bootstrap.cc:492] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc: Bootstrap starting.
I20260620 07:10:09.079927 13415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b03b5f7c444bc8aeca389992d60c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:09.080801 13867 tablet_bootstrap.cc:654] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:09.081037 13868 tablet_bootstrap.cc:654] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:09.082827 13867 tablet_bootstrap.cc:492] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec: No bootstrap required, opened a new log
I20260620 07:10:09.082979 13867 ts_tablet_manager.cc:1397] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 07:10:09.083810 13867 raft_consensus.cc:348] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:09.084049 13867 raft_consensus.cc:374] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:09.084134 13867 raft_consensus.cc:729] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2ec70afce24c1099d2f07d015010ec, State: Initialized, Role: FOLLOWER
I20260620 07:10:09.084373 13867 consensus_queue.cc:260] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:09.085158 13867 ts_tablet_manager.cc:1428] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 07:10:09.086329 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.085371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.67:51849)
0620 07:10:09.085929 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.086048 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":258,"threads_started":1}
I20260620 07:10:09.086813 13868 tablet_bootstrap.cc:492] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc: No bootstrap required, opened a new log
I20260620 07:10:09.086999 13868 ts_tablet_manager.cc:1397] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 07:10:09.088303 13868 raft_consensus.cc:348] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:09.088589 13868 raft_consensus.cc:374] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:09.088692 13868 raft_consensus.cc:729] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42141b2ca9ed4593a530955665d7a0fc, State: Initialized, Role: FOLLOWER
I20260620 07:10:09.089190 13868 consensus_queue.cc:260] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:09.089623 13772 heartbeater.cc:503] Master 127.6.53.125:38109 was elected leader, sending a full tablet report...
I20260620 07:10:09.092590 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.090531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.67:41809)
0620 07:10:09.092421 (+ 1890us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.092487 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1444}
I20260620 07:10:09.093448 13868 ts_tablet_manager.cc:1428] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260620 07:10:09.094123 13873 tablet_bootstrap.cc:492] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc: Bootstrap starting.
I20260620 07:10:09.094751 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.093884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.66:32777)
0620 07:10:09.094385 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.094486 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"threads_started":1}
I20260620 07:10:09.096076 13873 tablet_bootstrap.cc:654] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:09.096525 13631 heartbeater.cc:503] Master 127.6.53.125:38109 was elected leader, sending a full tablet report...
I20260620 07:10:09.096716 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.095560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.66:34271)
0620 07:10:09.096335 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.096432 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260620 07:10:09.098759 13873 tablet_bootstrap.cc:492] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc: No bootstrap required, opened a new log
I20260620 07:10:09.098917 13873 ts_tablet_manager.cc:1397] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 07:10:09.099591 13873 raft_consensus.cc:348] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:09.099794 13873 raft_consensus.cc:374] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:09.099929 13873 raft_consensus.cc:729] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659e9bb195db4166b7c0c6bc30452fbc, State: Initialized, Role: FOLLOWER
I20260620 07:10:09.100209 13873 consensus_queue.cc:260] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:09.100970 13873 ts_tablet_manager.cc:1428] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:10:09.102150 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.101143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.65:53417)
0620 07:10:09.101705 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.101862 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":263,"threads_started":1}
I20260620 07:10:09.103101 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.102797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.65:47953)
0620 07:10:09.102926 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.102993 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 07:10:09.103264 13490 heartbeater.cc:503] Master 127.6.53.125:38109 was elected leader, sending a full tablet report...
I20260620 07:10:09.176532 13874 raft_consensus.cc:482] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:10:09.176728 13874 raft_consensus.cc:504] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:09.177541 13874 leader_election.cc:288] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 659e9bb195db4166b7c0c6bc30452fbc (127.6.53.65:42627), ca2ec70afce24c1099d2f07d015010ec (127.6.53.67:42697)
I20260620 07:10:09.178105 13435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b03b5f7c444bc8aeca389992d60c4e" candidate_uuid: "42141b2ca9ed4593a530955665d7a0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659e9bb195db4166b7c0c6bc30452fbc" is_pre_election: true
I20260620 07:10:09.178139 13725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b03b5f7c444bc8aeca389992d60c4e" candidate_uuid: "42141b2ca9ed4593a530955665d7a0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca2ec70afce24c1099d2f07d015010ec" is_pre_election: true
I20260620 07:10:09.178393 13435 raft_consensus.cc:2454] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42141b2ca9ed4593a530955665d7a0fc in term 0.
I20260620 07:10:09.178452 13725 raft_consensus.cc:2454] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42141b2ca9ed4593a530955665d7a0fc in term 0.
I20260620 07:10:09.178870 13517 leader_election.cc:302] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42141b2ca9ed4593a530955665d7a0fc, 659e9bb195db4166b7c0c6bc30452fbc; no voters:
I20260620 07:10:09.179137 13874 raft_consensus.cc:2781] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:10:09.179229 13874 raft_consensus.cc:482] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:10:09.179288 13874 raft_consensus.cc:3037] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:09.181222 13874 raft_consensus.cc:504] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:09.181753 13874 leader_election.cc:288] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [CANDIDATE]: Term 1 election: Requested vote from peers 659e9bb195db4166b7c0c6bc30452fbc (127.6.53.65:42627), ca2ec70afce24c1099d2f07d015010ec (127.6.53.67:42697)
I20260620 07:10:09.182164 13435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b03b5f7c444bc8aeca389992d60c4e" candidate_uuid: "42141b2ca9ed4593a530955665d7a0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659e9bb195db4166b7c0c6bc30452fbc"
I20260620 07:10:09.182283 13725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b03b5f7c444bc8aeca389992d60c4e" candidate_uuid: "42141b2ca9ed4593a530955665d7a0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca2ec70afce24c1099d2f07d015010ec"
I20260620 07:10:09.182410 13435 raft_consensus.cc:3037] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:09.182533 13725 raft_consensus.cc:3037] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:09.184356 13435 raft_consensus.cc:2454] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42141b2ca9ed4593a530955665d7a0fc in term 1.
I20260620 07:10:09.184373 13725 raft_consensus.cc:2454] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42141b2ca9ed4593a530955665d7a0fc in term 1.
I20260620 07:10:09.184816 13517 leader_election.cc:302] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 42141b2ca9ed4593a530955665d7a0fc, 659e9bb195db4166b7c0c6bc30452fbc, ca2ec70afce24c1099d2f07d015010ec; no voters:
I20260620 07:10:09.185089 13874 raft_consensus.cc:2781] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:10:09.185453 13874 raft_consensus.cc:686] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [term 1 LEADER]: Becoming Leader. State: Replica: 42141b2ca9ed4593a530955665d7a0fc, State: Running, Role: LEADER
I20260620 07:10:09.185750 13874 consensus_queue.cc:237] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } } peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } }
I20260620 07:10:09.186659 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.186231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.66:42085)
0620 07:10:09.186426 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.186515 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 07:10:09.188810 13109 catalog_manager.cc:5495] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 42141b2ca9ed4593a530955665d7a0fc (127.6.53.66). New cstate: current_term: 1 leader_uuid: "42141b2ca9ed4593a530955665d7a0fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42141b2ca9ed4593a530955665d7a0fc" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 } health_report { overall_health: UNKNOWN } } }
I20260620 07:10:09.229128 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.224896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.6.53.66:39055)
0620 07:10:09.225093 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:09.225098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:09.225114 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:10:09.225243 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:09.225247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:09.225306 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:09.225375 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:10:09.225681 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.226438 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.227698 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.228087 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.228174 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:09.228511 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:10:09.228598 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:10:09.228659 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:09.228795 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 07:10:09.228849 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:10:09.231932 13725 raft_consensus.cc:1261] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec [term 1 FOLLOWER]: Refusing update from remote peer 42141b2ca9ed4593a530955665d7a0fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:10:09.231945 13435 raft_consensus.cc:1261] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc [term 1 FOLLOWER]: Refusing update from remote peer 42141b2ca9ed4593a530955665d7a0fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:10:09.232559 13879 consensus_queue.cc:1035] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "659e9bb195db4166b7c0c6bc30452fbc" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:09.232900 13874 consensus_queue.cc:1035] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca2ec70afce24c1099d2f07d015010ec" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:09.235246 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.234674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.67:36265)
0620 07:10:09.234950 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.235060 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260620 07:10:09.235718 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.235065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.65:56563)
0620 07:10:09.235324 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.235481 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260620 07:10:09.239820 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.239225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.66:40175)
0620 07:10:09.239544 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.239657 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232}
I20260620 07:10:09.240087 13882 mvcc.cc:205] Tried to move back new op lower bound from 7298823820208324608 to 7298823820027490304. Current Snapshot: MvccSnapshot[applied={T|T < 7298823820208324608}]
I20260620 07:10:09.241359 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.240155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.67:48091)
0620 07:10:09.241130 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.241230 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":873}
I20260620 07:10:09.249123 13109 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:10:09.249424 13109 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:10:09.250638 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.250056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.66:42409)
0620 07:10:09.250398 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.250485 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235}
I20260620 07:10:09.252822 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.252369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.6.53.65:49215)
0620 07:10:09.252558 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.252666 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 07:10:09.256006 13109 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0963332cf16b489e9eaf2607296f0e60: Sending DeleteTablet for 3 replicas of tablet 70b03b5f7c444bc8aeca389992d60c4e
I20260620 07:10:09.257131 13564 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b03b5f7c444bc8aeca389992d60c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:09 UTC) from {username='oryx', principal='oryx/127.6.53.125@KRBTEST.COM'} at 127.0.0.1:56712
I20260620 07:10:09.257397 13563 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b03b5f7c444bc8aeca389992d60c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:09 UTC) from {username='oryx', principal='oryx/127.6.53.125@KRBTEST.COM'} at 127.0.0.1:56712
I20260620 07:10:09.257776 13705 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b03b5f7c444bc8aeca389992d60c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:09 UTC) from {username='oryx', principal='oryx/127.6.53.125@KRBTEST.COM'} at 127.0.0.1:43778
I20260620 07:10:09.258499 13415 tablet_service.cc:1514] Processing DeleteTablet for tablet 70b03b5f7c444bc8aeca389992d60c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:09 UTC) from {username='oryx', principal='oryx/127.6.53.125@KRBTEST.COM'} at 127.0.0.1:47736
I20260620 07:10:09.258662 13892 tablet_replica.cc:330] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec: stopping tablet replica
I20260620 07:10:09.258831 13891 tablet_replica.cc:330] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc: stopping tablet replica
I20260620 07:10:09.258915 13892 raft_consensus.cc:2229] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:10:09.258989 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 13275
I20260620 07:10:09.259001 13891 raft_consensus.cc:2229] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:10:09.259188 13892 raft_consensus.cc:2258] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:09.259367 13891 raft_consensus.cc:2258] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:09.259931 13893 tablet_replica.cc:330] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc: stopping tablet replica
I20260620 07:10:09.260108 13893 raft_consensus.cc:2229] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:10:09.260392 13893 raft_consensus.cc:2258] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:09.261102 13891 ts_tablet_manager.cc:1905] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:09.262588 13893 ts_tablet_manager.cc:1905] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:09.262678 13892 ts_tablet_manager.cc:1905] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:09.267508 13891 ts_tablet_manager.cc:1918] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:10:09.267660 13891 log.cc:1196] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/70b03b5f7c444bc8aeca389992d60c4e
I20260620 07:10:09.268111 13891 ts_tablet_manager.cc:1939] T 70b03b5f7c444bc8aeca389992d60c4e P 42141b2ca9ed4593a530955665d7a0fc: Deleting consensus metadata
I20260620 07:10:09.269127 13093 catalog_manager.cc:4841] TS 42141b2ca9ed4593a530955665d7a0fc (127.6.53.66:39055): tablet 70b03b5f7c444bc8aeca389992d60c4e (table test-table [id=4f9ee5d3482047659808a1d246897eca]) successfully deleted
W20260620 07:10:09.269634 13093 catalog_manager.cc:4808] TS 42141b2ca9ed4593a530955665d7a0fc (127.6.53.66:39055): delete failed for tablet 70b03b5f7c444bc8aeca389992d60c4e because the tablet was not found. No further retry: Not found: Tablet not found: 70b03b5f7c444bc8aeca389992d60c4e
I20260620 07:10:09.269685 13893 ts_tablet_manager.cc:1918] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:10:09.269960 13893 log.cc:1196] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/70b03b5f7c444bc8aeca389992d60c4e
I20260620 07:10:09.270026 13892 ts_tablet_manager.cc:1918] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:10:09.270154 13892 log.cc:1196] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/70b03b5f7c444bc8aeca389992d60c4e
I20260620 07:10:09.270493 13893 ts_tablet_manager.cc:1939] T 70b03b5f7c444bc8aeca389992d60c4e P 659e9bb195db4166b7c0c6bc30452fbc: Deleting consensus metadata
I20260620 07:10:09.270588 13892 ts_tablet_manager.cc:1939] T 70b03b5f7c444bc8aeca389992d60c4e P ca2ec70afce24c1099d2f07d015010ec: Deleting consensus metadata
I20260620 07:10:09.271531 13093 catalog_manager.cc:4841] TS 659e9bb195db4166b7c0c6bc30452fbc (127.6.53.65:42627): tablet 70b03b5f7c444bc8aeca389992d60c4e (table test-table [id=4f9ee5d3482047659808a1d246897eca]) successfully deleted
I20260620 07:10:09.271678 13094 catalog_manager.cc:4841] TS ca2ec70afce24c1099d2f07d015010ec (127.6.53.67:42697): tablet 70b03b5f7c444bc8aeca389992d60c4e (table test-table [id=4f9ee5d3482047659808a1d246897eca]) successfully deleted
I20260620 07:10:09.275399 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.259798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47740 (local address 127.6.53.65:42627)
0620 07:10:09.260937 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:09.260943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:09.260960 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:10:09.261020 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:09.261025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:09.261103 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:09.261213 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:09.262767 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.263880 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.265583 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.266283 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.266396 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:09.269930 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:09.269942 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:09.269963 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:09.269996 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:09.273226 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:09.273753 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:09.273759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:09.273761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:09.273869 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:09.274160 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:09.274163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:09.274165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:09.274566 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:09.274770 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:09.275028 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 07:10:09.275182 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886}
I20260620 07:10:09.276021 13475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:09.366184 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 13275
I20260620 07:10:09.383982 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 13495
I20260620 07:10:09.397790 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.384661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.6.53.66:39055)
0620 07:10:09.384845 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:09.384851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:09.384967 (+ 116us) server_negotiation.cc:407] Connection header received
0620 07:10:09.385199 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:09.385203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:09.385277 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:09.385384 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:09.386664 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.387464 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.388646 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.389050 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.389177 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:09.391885 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:09.391898 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:09.391914 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:09.391950 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:09.395023 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:09.395708 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:09.395723 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:09.395726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:09.395851 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:09.396220 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:09.396227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:09.396230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:09.396604 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:09.396922 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:09.397202 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 07:10:09.397431 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:10:09.398491 13624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:09.455153 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.454615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.0.0.1:40036)
0620 07:10:09.454806 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.455001 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 07:10:09.456220 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.455860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41085 (local address 127.0.0.1:40038)
0620 07:10:09.456042 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:09.456117 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 07:10:09.516772 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 13495
I20260620 07:10:09.537878 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 13642
I20260620 07:10:09.550853 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:09.538658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43794 (local address 127.6.53.67:42697)
0620 07:10:09.538833 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:09.538841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:09.538863 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:10:09.539090 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:09.539095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:09.539168 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:09.539275 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:09.540608 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.541289 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.542481 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:09.542881 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:09.543010 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:09.545307 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:09.545319 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:09.545345 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:09.545376 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:09.548125 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:09.548777 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:09.548790 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:09.548792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:09.548905 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:09.549278 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:09.549285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:09.549288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:09.549669 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:09.549995 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:09.550305 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 07:10:09.550506 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:10:09.551505 13765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:09.641896 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 13642
I20260620 07:10:09.659574 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 12972
I20260620 07:10:09.660625 13139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:09.780471 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 12972
I20260620 07:10:09.801340 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 13072
I20260620 07:10:09.802354 13333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:09.920830 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 13072
2026-06-20T07:10:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9598 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:10:10 dist-test-slave-4n6h krb5kdc[13902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:10:10 dist-test-slave-4n6h krb5kdc[13902](info): set up 2 sockets
Jun 20 07:10:10 dist-test-slave-4n6h krb5kdc[13902](info): commencing operation
krb5kdc: starting...
W20260620 07:10:11.119709 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:10:11 dist-test-slave-4n6h krb5kdc[13902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:10:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:10:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.126.keytab.
Entry for principal oryx/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.126.keytab.
I20260620 07:10:11.387651 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:40513
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.126.keytab
--principal=oryx/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:40513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:11.722081 13915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:11.722378 13915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:11.722555 13915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:11.729562 13915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:10:11.729652 13915 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 07:10:11.729749 13915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:11.729872 13915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:11.729960 13915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:10:11.730072 13915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:10:11.739488 13915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:40513
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.126.keytab
--principal=oryx/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:40513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:11.740674 13915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:11.741957 13915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:11.753567 13921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:11.753539 13923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:11.753734 13920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:11.754329 13915 server_base.cc:1034] running on GCE node
I20260620 07:10:11.755069 13915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:11.756503 13915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:11.757753 13915 hybrid_clock.cc:648] HybridClock initialized: now 1781939411757673 us; error 53 us; skew 500 ppm
I20260620 07:10:11.758140 13915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:11 dist-test-slave-4n6h krb5kdc[13902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939411, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:11.761484 13915 init.cc:377] Logged in from keytab as oryx/127.6.53.126@KRBTEST.COM (short username oryx)
I20260620 07:10:11.762933 13915 webserver.cc:460] Webserver started at http://127.6.53.126:34013/ using document root <none> and password file <none>
I20260620 07:10:11.763593 13915 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:11.763681 13915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:11.763973 13915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:11.766973 13915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "23ad33659ad74a069fecb747845133f3"
format_stamp: "Formatted at 2026-06-20 07:10:11 on dist-test-slave-4n6h"
server_key: "8a2993094cea6ddf0844a61f3da9cd23"
server_key_iv: "b970e7d94bea7c100b940e65416ef9aa"
server_key_version: "encryptionkey@0"
I20260620 07:10:11.767750 13915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "23ad33659ad74a069fecb747845133f3"
format_stamp: "Formatted at 2026-06-20 07:10:11 on dist-test-slave-4n6h"
server_key: "8a2993094cea6ddf0844a61f3da9cd23"
server_key_iv: "b970e7d94bea7c100b940e65416ef9aa"
server_key_version: "encryptionkey@0"
I20260620 07:10:11.772830 13915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:10:11.776273 13930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:11.777571 13915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:10:11.777760 13915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "23ad33659ad74a069fecb747845133f3"
format_stamp: "Formatted at 2026-06-20 07:10:11 on dist-test-slave-4n6h"
server_key: "8a2993094cea6ddf0844a61f3da9cd23"
server_key_iv: "b970e7d94bea7c100b940e65416ef9aa"
server_key_version: "encryptionkey@0"
I20260620 07:10:11.777989 13915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:11.792800 13915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:11.797274 13915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:11.797562 13915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:11.819468 13915 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:40513
I20260620 07:10:11.819499 13991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:40513 every 8 connection(s)
I20260620 07:10:11.820864 13915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:10:11.824314 13992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:11.828770 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 13915
I20260620 07:10:11.828975 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:10:11.829778 6357 external_mini_cluster.cc:1428] Setting key a003b92366c047f5226e8c351783e709
I20260620 07:10:11.835633 13992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3: Bootstrap starting.
I20260620 07:10:11.839823 13992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:11.841042 13992 log.cc:824] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:11.844224 13992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3: No bootstrap required, opened a new log
Jun 20 07:10:11 dist-test-slave-4n6h krb5kdc[13902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM
I20260620 07:10:11.854707 13992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ad33659ad74a069fecb747845133f3" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40513 } }
I20260620 07:10:11.855163 13992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:11.855273 13992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ad33659ad74a069fecb747845133f3, State: Initialized, Role: FOLLOWER
I20260620 07:10:11.855872 13992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ad33659ad74a069fecb747845133f3" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40513 } }
I20260620 07:10:11.856171 13992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:10:11.856324 13992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:10:11.856496 13992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:11.856997 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:11.831809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.6.53.126:40513)
0620 07:10:11.832611 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:11.832621 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:11.832658 (+ 37us) server_negotiation.cc:407] Connection header received
0620 07:10:11.834405 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:11.834425 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:11.834921 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:11.835358 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:11.837474 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:11.839758 (+ 2284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:11.841648 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:11.842771 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:11.842989 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:11.848394 (+ 5405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:11.848411 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:11.848427 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:11.848472 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:11.852416 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:11.853098 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:11.853107 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:11.853118 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:11.853317 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:11.853795 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:11.853801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:11.853805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:11.854486 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:11.854782 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:11.855198 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 07:10:11.855623 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":412,"threads_started":1}
I20260620 07:10:11.859040 13992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ad33659ad74a069fecb747845133f3" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40513 } }
I20260620 07:10:11.859531 13992 leader_election.cc:302] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23ad33659ad74a069fecb747845133f3; no voters:
I20260620 07:10:11.860046 13992 leader_election.cc:288] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:10:11.860167 13997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:10:11.860888 13997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [term 1 LEADER]: Becoming Leader. State: Replica: 23ad33659ad74a069fecb747845133f3, State: Running, Role: LEADER
I20260620 07:10:11.861433 13997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ad33659ad74a069fecb747845133f3" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40513 } }
I20260620 07:10:11.862115 13992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:10:11.864066 13998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23ad33659ad74a069fecb747845133f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ad33659ad74a069fecb747845133f3" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40513 } } }
I20260620 07:10:11.864388 13998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:11.864478 13999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23ad33659ad74a069fecb747845133f3. Latest consensus state: current_term: 1 leader_uuid: "23ad33659ad74a069fecb747845133f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ad33659ad74a069fecb747845133f3" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40513 } } }
I20260620 07:10:11.864779 13999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3 [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:11.869813 14002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:10:11.874034 14002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:10:11.881889 14002 catalog_manager.cc:1347] Generated new cluster ID: 119f205b4f884d8bbd6773247e93a676
I20260620 07:10:11.881994 14002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:10:11.892362 14002 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:10:11.894219 14002 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:10:11.903625 14002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3: Generated new TSK 0
I20260620 07:10:11.904546 14002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.65.keytab.
Entry for principal oryx/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.65.keytab.
I20260620 07:10:11.956247 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40513
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.65.keytab
--principal=oryx/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:12.271476 14018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:12.271777 14018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:12.271955 14018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:12.279258 14018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:12.279403 14018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:12.279549 14018 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:10:12.280344 14018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:10:12.293363 14018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.65.keytab
--principal=oryx/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.53.126:40513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:12.294881 14018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:12.296265 14018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:12.309751 14023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:12.309935 14026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:12.310426 14024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:12.311372 14018 server_base.cc:1034] running on GCE node
I20260620 07:10:12.312292 14018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:12.313476 14018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:12.314785 14018 hybrid_clock.cc:648] HybridClock initialized: now 1781939412314695 us; error 77 us; skew 500 ppm
I20260620 07:10:12.315392 14018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:12 dist-test-slave-4n6h krb5kdc[13902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939412, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:12.319273 14018 init.cc:377] Logged in from keytab as oryx/127.6.53.65@KRBTEST.COM (short username oryx)
I20260620 07:10:12.321378 14018 webserver.cc:460] Webserver started at http://127.6.53.65:38927/ using document root <none> and password file <none>
I20260620 07:10:12.322393 14018 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:12.322526 14018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:12.322862 14018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:12.325692 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "207fa9f0521d483aa525afb882a2b6b6"
format_stamp: "Formatted at 2026-06-20 07:10:12 on dist-test-slave-4n6h"
server_key: "407a77edc7f8c276a6c31f458eec7d36"
server_key_iv: "cb1032a6185108193d4c8c31e68a8643"
server_key_version: "encryptionkey@0"
I20260620 07:10:12.326570 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "207fa9f0521d483aa525afb882a2b6b6"
format_stamp: "Formatted at 2026-06-20 07:10:12 on dist-test-slave-4n6h"
server_key: "407a77edc7f8c276a6c31f458eec7d36"
server_key_iv: "cb1032a6185108193d4c8c31e68a8643"
server_key_version: "encryptionkey@0"
I20260620 07:10:12.331844 14018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 07:10:12.336395 14033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:12.337992 14018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:10:12.338198 14018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "207fa9f0521d483aa525afb882a2b6b6"
format_stamp: "Formatted at 2026-06-20 07:10:12 on dist-test-slave-4n6h"
server_key: "407a77edc7f8c276a6c31f458eec7d36"
server_key_iv: "cb1032a6185108193d4c8c31e68a8643"
server_key_version: "encryptionkey@0"
I20260620 07:10:12.338512 14018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:12.359093 14018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:12.364609 14018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:12.364985 14018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:12.368929 14018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:12.369040 14018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:12.369351 14018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:12.369475 14018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:10:12 dist-test-slave-4n6h krb5kdc[13902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939412, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM
I20260620 07:10:12.400372 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:12.369903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:44919 (local address 127.6.53.126:40513)
0620 07:10:12.370842 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:12.370851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:12.371782 (+ 931us) server_negotiation.cc:407] Connection header received
0620 07:10:12.374306 (+ 2524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:12.374313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:12.374681 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:12.374926 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:12.378388 (+ 3462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:12.379852 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:12.383074 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:12.383708 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:12.383880 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:12.389563 (+ 5683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:12.389574 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:12.389582 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:12.389633 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:12.396186 (+ 6553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:12.397335 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:12.397344 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:12.397351 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:12.397522 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:12.398104 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:12.398109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:12.398112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:12.398549 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:12.398708 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:12.399725 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0620 07:10:12.399972 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":627,"threads_started":1}
I20260620 07:10:12.402108 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:12.370400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:40513 (local address 127.6.53.65:44919)
0620 07:10:12.371534 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:12.371578 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:12.373937 (+ 2359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:12.375758 (+ 1821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:12.375775 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:12.376713 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:12.377995 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:12.378028 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:12.380192 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:12.380200 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:12.382874 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:12.382892 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:12.384016 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:12.384028 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:12.384402 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:12.385409 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:12.385460 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:12.389219 (+ 3759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:12.396416 (+ 7197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:12.396429 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:12.396451 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:12.397146 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:12.397693 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:12.397702 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:12.397706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:12.397974 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:12.398806 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:12.398814 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:12.399295 (+ 481us) client_negotiation.cc:769] Sending connection context
0620 07:10:12.399630 (+ 335us) client_negotiation.cc:240] Negotiation successful
0620 07:10:12.400186 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":456,"threads_started":1}
I20260620 07:10:12.427450 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:12.409676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:40513 (local address 127.6.53.65:50843)
0620 07:10:12.410728 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:12.410754 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:12.411065 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:12.411464 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:12.411469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:12.412454 (+ 985us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:12.413006 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:12.413026 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:12.414507 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:12.414511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:12.417141 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:12.417155 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:12.418279 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:12.418289 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:12.418485 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:12.419424 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:12.419446 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:12.420785 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:12.424199 (+ 3414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:12.424204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:12.424208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:12.424854 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:12.425310 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:12.425319 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:12.425324 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:12.425490 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:12.426237 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:12.426241 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:12.426540 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 07:10:12.426748 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 07:10:12.427067 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":650,"threads_started":1}
I20260620 07:10:12.428207 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:12.409620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:50843 (local address 127.6.53.126:40513)
0620 07:10:12.409804 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:12.409810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:12.410876 (+ 1066us) server_negotiation.cc:407] Connection header received
0620 07:10:12.411229 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:12.411233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:12.411293 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:12.411440 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:12.413239 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:12.414310 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:12.417309 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:12.418003 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:12.418171 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:12.421044 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:12.421050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:12.421053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:12.421081 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:12.424065 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:12.424983 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:12.424989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:12.424996 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:12.425165 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:12.425616 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:12.425622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:12.425626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:12.426020 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:12.426174 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:12.427034 (+ 860us) server_negotiation.cc:299] Negotiation successful
0620 07:10:12.427513 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:10:12.434166 14018 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:43155
I20260620 07:10:12.434224 14153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:43155 every 8 connection(s)
I20260620 07:10:12.435490 14018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:10:12.438098 14154 heartbeater.cc:347] Connected to a master server at 127.6.53.126:40513
I20260620 07:10:12.438426 14154 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:12.439103 14154 heartbeater.cc:511] Master 127.6.53.126:40513 requested a full tablet report, sending...
I20260620 07:10:12.440191 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 14018
I20260620 07:10:12.440429 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:10:12.441061 6357 external_mini_cluster.cc:1428] Setting key 6a505dc7edd2e85c8ce9356fa4c6571c
I20260620 07:10:12.441129 13947 ts_manager.cc:194] Registered new tserver with Master: 207fa9f0521d483aa525afb882a2b6b6 (127.6.53.65:43155)
I20260620 07:10:12.442926 13947 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:44919
WARNING: no policy specified for oryx/127.6.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.66.keytab.
Entry for principal oryx/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.66.keytab.
I20260620 07:10:12.502353 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40513
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.66.keytab
--principal=oryx/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:12.815616 14158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:12.815877 14158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:12.816042 14158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:12.823143 14158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:12.823242 14158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:12.823402 14158 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:10:12.823513 14158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:10:12.833279 14158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.66.keytab
--principal=oryx/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.53.126:40513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:12.834496 14158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:12.835745 14158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:12.848562 14166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:12.848662 14163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:12.848631 14164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:12.849601 14158 server_base.cc:1034] running on GCE node
I20260620 07:10:12.850207 14158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:12.851202 14158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:12.852422 14158 hybrid_clock.cc:648] HybridClock initialized: now 1781939412852366 us; error 35 us; skew 500 ppm
I20260620 07:10:12.852711 14158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:12 dist-test-slave-4n6h krb5kdc[13902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939412, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:12.855805 14158 init.cc:377] Logged in from keytab as oryx/127.6.53.66@KRBTEST.COM (short username oryx)
I20260620 07:10:12.857081 14158 webserver.cc:460] Webserver started at http://127.6.53.66:33709/ using document root <none> and password file <none>
I20260620 07:10:12.857748 14158 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:12.857853 14158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:12.858186 14158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:12.861146 14158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "1dd9e6b7b49f43abaa81b235d451959b"
format_stamp: "Formatted at 2026-06-20 07:10:12 on dist-test-slave-4n6h"
server_key: "6d0208fb4ad2fad4aac2ce9370140b78"
server_key_iv: "225e9dd561ce6de774a0138d4146d396"
server_key_version: "encryptionkey@0"
I20260620 07:10:12.861912 14158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "1dd9e6b7b49f43abaa81b235d451959b"
format_stamp: "Formatted at 2026-06-20 07:10:12 on dist-test-slave-4n6h"
server_key: "6d0208fb4ad2fad4aac2ce9370140b78"
server_key_iv: "225e9dd561ce6de774a0138d4146d396"
server_key_version: "encryptionkey@0"
I20260620 07:10:12.866915 14158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 07:10:12.870365 14173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:12.871695 14158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:10:12.871862 14158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "1dd9e6b7b49f43abaa81b235d451959b"
format_stamp: "Formatted at 2026-06-20 07:10:12 on dist-test-slave-4n6h"
server_key: "6d0208fb4ad2fad4aac2ce9370140b78"
server_key_iv: "225e9dd561ce6de774a0138d4146d396"
server_key_version: "encryptionkey@0"
I20260620 07:10:12.872066 14158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:12.895064 14158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:12.899020 14158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:12.899289 14158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:12.901847 14158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:12.901959 14158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:12.902110 14158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:12.902199 14158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:10:12 dist-test-slave-4n6h krb5kdc[13902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939412, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM
I20260620 07:10:12.929596 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:12.903501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:40513 (local address 127.6.53.66:42725)
0620 07:10:12.904390 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:12.904442 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:12.906736 (+ 2294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:12.907577 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:12.907592 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:12.908534 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:12.909650 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:12.909682 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:12.911110 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:12.911119 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:12.913955 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:12.913979 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:12.915016 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:12.915026 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:12.915376 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:12.916107 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:12.916150 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:12.919952 (+ 3802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:12.923910 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:12.923921 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:12.923942 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:12.924681 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:12.925062 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:12.925071 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:12.925073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:12.925366 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:12.926012 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:12.926022 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:12.926612 (+ 590us) client_negotiation.cc:769] Sending connection context
0620 07:10:12.926999 (+ 387us) client_negotiation.cc:240] Negotiation successful
0620 07:10:12.927689 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":406,"threads_started":1}
I20260620 07:10:12.930781 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:12.902531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:42725 (local address 127.6.53.126:40513)
0620 07:10:12.902802 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:12.902811 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:12.904574 (+ 1763us) server_negotiation.cc:407] Connection header received
0620 07:10:12.907036 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:12.907042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:12.907131 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:12.907237 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:12.909888 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:12.910916 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:12.914133 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:12.914731 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:12.914871 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:12.920334 (+ 5463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:12.920347 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:12.920355 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:12.920394 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:12.923725 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:12.924811 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:12.924815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:12.924817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:12.924927 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:12.925525 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:12.925531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:12.925534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:12.925828 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:12.925997 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:12.930036 (+ 4039us) server_negotiation.cc:299] Negotiation successful
0620 07:10:12.930313 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 07:10:12.947223 14158 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:39999
I20260620 07:10:12.947304 14292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:39999 every 8 connection(s)
I20260620 07:10:12.948376 14158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:10:12.950347 14293 heartbeater.cc:347] Connected to a master server at 127.6.53.126:40513
I20260620 07:10:12.950579 14293 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:12.951160 14293 heartbeater.cc:511] Master 127.6.53.126:40513 requested a full tablet report, sending...
I20260620 07:10:12.952447 13947 ts_manager.cc:194] Registered new tserver with Master: 1dd9e6b7b49f43abaa81b235d451959b (127.6.53.66:39999)
I20260620 07:10:12.952450 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:12.936244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:41243 (local address 127.6.53.126:40513)
0620 07:10:12.936468 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:12.936476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:12.937342 (+ 866us) server_negotiation.cc:407] Connection header received
0620 07:10:12.937460 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:12.937465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:12.937544 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:12.937654 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:12.938845 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:12.940022 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:12.942519 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:12.943334 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:12.943514 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:12.945758 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:12.945763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:12.945766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:12.945792 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:12.948970 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:12.950342 (+ 1372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:12.950349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:12.950353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:12.950537 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:12.950915 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:12.950920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:12.950923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:12.951234 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:12.951459 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:12.951905 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 07:10:12.952113 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 07:10:12.952540 14265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:12.936304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:40513 (local address 127.6.53.66:41243)
0620 07:10:12.937260 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:12.937275 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:12.937380 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:12.937700 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:12.937706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:12.938396 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:12.938699 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:12.938711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:12.940168 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:12.940176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:12.942361 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:12.942382 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:12.943565 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:12.943581 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:12.943727 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:12.944353 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:12.944378 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:12.945596 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:12.949121 (+ 3525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:12.949129 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:12.949136 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:12.949827 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:12.950647 (+ 820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:12.950652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:12.950660 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:12.950786 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:12.951430 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:12.951435 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:12.951622 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 07:10:12.951810 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 07:10:12.952095 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":529,"threads_started":1}
I20260620 07:10:12.952971 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 14158
I20260620 07:10:12.953105 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:10:12.953434 13947 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:42725
I20260620 07:10:12.953477 6357 external_mini_cluster.cc:1428] Setting key 472822d160f8d0fe80e8e4b95a3e2152
WARNING: no policy specified for oryx/127.6.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.67.keytab.
Entry for principal oryx/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.67.keytab.
I20260620 07:10:12.997673 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40513
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.67.keytab
--principal=oryx/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:13.317749 14297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:13.317991 14297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:13.318100 14297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:13.325021 14297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:13.325112 14297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:13.325199 14297 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:10:13.325277 14297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:10:13.334771 14297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.67.keytab
--principal=oryx/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.53.126:40513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:13.335868 14297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:13.337060 14297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:13.349987 14302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:13.349961 14303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:13.349987 14305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:13.351157 14297 server_base.cc:1034] running on GCE node
I20260620 07:10:13.351750 14297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:13.352636 14297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:13.353880 14297 hybrid_clock.cc:648] HybridClock initialized: now 1781939413353814 us; error 44 us; skew 500 ppm
I20260620 07:10:13.354187 14297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:13 dist-test-slave-4n6h krb5kdc[13902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939413, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:13.357240 14297 init.cc:377] Logged in from keytab as oryx/127.6.53.67@KRBTEST.COM (short username oryx)
I20260620 07:10:13.358551 14297 webserver.cc:460] Webserver started at http://127.6.53.67:44687/ using document root <none> and password file <none>
I20260620 07:10:13.359144 14297 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:13.359232 14297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:13.359470 14297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:13.362237 14297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "09f0514fc0604b3db4b4385605d73cee"
format_stamp: "Formatted at 2026-06-20 07:10:13 on dist-test-slave-4n6h"
server_key: "f88f4ace929d6d82510cf6895b5a3590"
server_key_iv: "85e1a46ab2e8c927ca27b4a8d9cfa718"
server_key_version: "encryptionkey@0"
I20260620 07:10:13.363042 14297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "09f0514fc0604b3db4b4385605d73cee"
format_stamp: "Formatted at 2026-06-20 07:10:13 on dist-test-slave-4n6h"
server_key: "f88f4ace929d6d82510cf6895b5a3590"
server_key_iv: "85e1a46ab2e8c927ca27b4a8d9cfa718"
server_key_version: "encryptionkey@0"
I20260620 07:10:13.368319 14297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:10:13.371549 14312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:13.372777 14297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:10:13.372963 14297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "09f0514fc0604b3db4b4385605d73cee"
format_stamp: "Formatted at 2026-06-20 07:10:13 on dist-test-slave-4n6h"
server_key: "f88f4ace929d6d82510cf6895b5a3590"
server_key_iv: "85e1a46ab2e8c927ca27b4a8d9cfa718"
server_key_version: "encryptionkey@0"
I20260620 07:10:13.373171 14297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:13.423934 14297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:13.427932 14297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:13.428200 14297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:13.430716 14297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:13.430833 14297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:13.430974 14297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:13.431046 14297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:13.445443 14154 heartbeater.cc:503] Master 127.6.53.126:40513 was elected leader, sending a full tablet report...
Jun 20 07:10:13 dist-test-slave-4n6h krb5kdc[13902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939413, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM
I20260620 07:10:13.457013 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:13.432139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:40513 (local address 127.6.53.67:36977)
0620 07:10:13.432885 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:13.432919 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:13.435669 (+ 2750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:13.436639 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:13.436650 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:13.437403 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:13.438156 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:13.438178 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.439682 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.439686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:13.442081 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:13.442100 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.443022 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.443034 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:13.443373 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.444030 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:13.444055 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:13.447845 (+ 3790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:13.451884 (+ 4039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:13.451901 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:13.451926 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:13.452673 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:13.453135 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:13.453142 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:13.453145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:13.453491 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:13.454229 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:13.454241 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:13.454705 (+ 464us) client_negotiation.cc:769] Sending connection context
0620 07:10:13.455071 (+ 366us) client_negotiation.cc:240] Negotiation successful
0620 07:10:13.455618 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":355,"threads_started":1}
I20260620 07:10:13.458513 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:13.431503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:36977 (local address 127.6.53.126:40513)
0620 07:10:13.431690 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:13.431697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:13.433036 (+ 1339us) server_negotiation.cc:407] Connection header received
0620 07:10:13.435978 (+ 2942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:13.435984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:13.436064 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:13.436133 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:13.438402 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.439490 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.442230 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.442698 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.442836 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.448048 (+ 5212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:13.448054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:13.448058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:13.448093 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:13.451716 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:13.452837 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:13.452845 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:13.452851 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:13.453006 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:13.453645 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:13.453653 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:13.453658 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:13.453953 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:13.454138 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:13.457850 (+ 3712us) server_negotiation.cc:299] Negotiation successful
0620 07:10:13.458124 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:10:13.475901 14297 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:37775
I20260620 07:10:13.475925 14431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:37775 every 8 connection(s)
I20260620 07:10:13.477396 14297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:10:13.479628 14432 heartbeater.cc:347] Connected to a master server at 127.6.53.126:40513
I20260620 07:10:13.479848 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:13.464166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:49771 (local address 127.6.53.126:40513)
0620 07:10:13.464326 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:13.464331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:13.465202 (+ 871us) server_negotiation.cc:407] Connection header received
0620 07:10:13.465472 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:13.465479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:13.465564 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:13.465629 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:13.466937 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.467846 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.470508 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.471103 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.471253 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.473706 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:13.473711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:13.473714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:13.473740 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:13.476813 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:13.477580 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:13.477586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:13.477589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:13.477729 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:13.478284 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:13.478290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:13.478293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:13.478643 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:13.478791 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:13.479279 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 07:10:13.479493 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:10:13.479887 14432 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:13.479983 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:13.464272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:40513 (local address 127.6.53.67:49771)
0620 07:10:13.465083 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:13.465098 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:13.465321 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:13.465807 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:13.465813 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:13.466472 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:13.466789 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:13.466802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.467988 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.467996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:13.470352 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:13.470367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.471301 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.471308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:13.471493 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.472232 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:13.472262 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:13.473543 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:13.476942 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:13.476945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:13.476948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:13.477410 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:13.477863 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:13.477889 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:13.477894 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:13.478070 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:13.478863 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:13.478868 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:13.479033 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 07:10:13.479174 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 07:10:13.479458 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":437,"threads_started":1}
I20260620 07:10:13.480659 14432 heartbeater.cc:511] Master 127.6.53.126:40513 requested a full tablet report, sending...
I20260620 07:10:13.481987 13946 ts_manager.cc:194] Registered new tserver with Master: 09f0514fc0604b3db4b4385605d73cee (127.6.53.67:37775)
I20260620 07:10:13.482959 13946 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:36977
I20260620 07:10:13.486757 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 14297
I20260620 07:10:13.486938 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:10:13.487391 6357 external_mini_cluster.cc:1428] Setting key d2a560e4b8b747a87b26dca371701fba
I20260620 07:10:13.489897 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 07:10:13 dist-test-slave-4n6h krb5kdc[13902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939413, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:10:13 dist-test-slave-4n6h krb5kdc[13902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:13 dist-test-slave-4n6h krb5kdc[13902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 07:10:13.535485 14042 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:13.525529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.6.53.126:40513)
0620 07:10:13.525751 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:13.525757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:13.525958 (+ 201us) server_negotiation.cc:407] Connection header received
0620 07:10:13.526164 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:13.526175 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:13.526236 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:13.526425 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:13.527721 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.528426 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.529648 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.530048 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.530197 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.535237 (+ 5040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32970: BlockingRecv error: recv got EOF from 127.0.0.1:32970 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 07:10:13 dist-test-slave-4n6h krb5kdc[13902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939413, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM
I20260620 07:10:13.550493 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:13.538144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32986 (local address 127.6.53.126:40513)
0620 07:10:13.538336 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:13.538341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:13.538682 (+ 341us) server_negotiation.cc:407] Connection header received
0620 07:10:13.538789 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:13.538792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:13.538870 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:13.538946 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:13.539941 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.540590 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.541745 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.542159 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.542285 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.545580 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:13.545585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:13.545588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:13.545615 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:13.548091 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:13.548702 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:13.548712 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:13.548715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:13.548832 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:13.549180 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:13.549186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:13.549189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:13.549447 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:13.549741 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:13.549968 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 07:10:13.550179 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:10:13.554086 13946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:10:13.561313 13946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:10:13.588371 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:13.573497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:37775 (local address 127.0.0.1:34912)
0620 07:10:13.576606 (+ 3109us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:13.576642 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:13.576878 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:13.577962 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:13.577972 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:13.578010 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:13.578566 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:13.578582 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.581409 (+ 2827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.581417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:13.584537 (+ 3120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:13.584555 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.587350 (+ 2795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.587356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:13.587541 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.587587 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 07:10:13.587711 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 07:10:13.587934 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2926,"mutex_wait_us":1,"thread_start_us":419,"threads_started":1}
I20260620 07:10:13.588835 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:13.574220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34912 (local address 127.6.53.67:37775)
0620 07:10:13.576387 (+ 2167us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:13.576398 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:13.576769 (+ 371us) server_negotiation.cc:407] Connection header received
0620 07:10:13.577090 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:13.577101 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:13.577475 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:13.577683 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:13.579042 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.581214 (+ 2172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.584762 (+ 3548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.586760 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.586921 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.587201 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:13.588363 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0620 07:10:13.588539 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1928,"thread_start_us":1728,"threads_started":1}
I20260620 07:10:13.590016 14450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:13.573090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:39999 (local address 127.0.0.1:42452)
0620 07:10:13.576606 (+ 3516us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:13.576640 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:13.576807 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:13.579169 (+ 2362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:13.579177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:13.579211 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:13.579743 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:13.579757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.583337 (+ 3580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.583345 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:13.587491 (+ 4146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:13.587510 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.589223 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.589235 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:13.589452 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.589482 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:10:13.589573 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 07:10:13.589698 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3227,"thread_start_us":3059,"threads_started":1}
I20260620 07:10:13.590080 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:13.573240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42452 (local address 127.6.53.66:39999)
0620 07:10:13.575955 (+ 2715us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:13.575967 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:13.577022 (+ 1055us) server_negotiation.cc:407] Connection header received
0620 07:10:13.577142 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:13.577150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:13.578707 (+ 1557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:13.578923 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:13.579955 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.583147 (+ 3192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.587724 (+ 4577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.588997 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.589118 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.589319 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:13.589618 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 07:10:13.589778 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2507,"thread_start_us":2333,"threads_started":1}
I20260620 07:10:13.592717 14362 tablet_service.cc:1467] Processing CreateTablet for tablet 88c9ba8d0138408387b1ab2d583b266d (DEFAULT_TABLE table=test-table [id=e0a5093bacc941578ee22c38e7156d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:13.593117 14223 tablet_service.cc:1467] Processing CreateTablet for tablet 88c9ba8d0138408387b1ab2d583b266d (DEFAULT_TABLE table=test-table [id=e0a5093bacc941578ee22c38e7156d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:13.594152 14362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c9ba8d0138408387b1ab2d583b266d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:13.594164 14223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c9ba8d0138408387b1ab2d583b266d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:13.595810 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:13.576695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:43155 (local address 127.0.0.1:56568)
0620 07:10:13.588022 (+ 11327us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:13.588042 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:13.588161 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:13.588891 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:13.588895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:13.588920 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:13.589330 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:13.589345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.591346 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.591352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:13.593262 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:13.593276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.595038 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.595044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:13.595215 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.595245 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:10:13.595343 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 07:10:13.595463 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11223,"mutex_wait_us":39,"thread_start_us":11094,"threads_started":1}
I20260620 07:10:13.595852 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:13.576780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56568 (local address 127.6.53.65:43155)
0620 07:10:13.587495 (+ 10715us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:13.587504 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:13.588161 (+ 657us) server_negotiation.cc:407] Connection header received
0620 07:10:13.588284 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:13.588290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:13.588508 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:13.588682 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:13.589526 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.591184 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.593460 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.594839 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.594974 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.595144 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:13.595410 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 07:10:13.595567 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10563,"thread_start_us":10721,"threads_started":1}
I20260620 07:10:13.599260 14084 tablet_service.cc:1467] Processing CreateTablet for tablet 88c9ba8d0138408387b1ab2d583b266d (DEFAULT_TABLE table=test-table [id=e0a5093bacc941578ee22c38e7156d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:13.600461 14084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88c9ba8d0138408387b1ab2d583b266d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:13.605580 14456 tablet_bootstrap.cc:492] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b: Bootstrap starting.
I20260620 07:10:13.607751 14457 tablet_bootstrap.cc:492] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee: Bootstrap starting.
I20260620 07:10:13.609018 14456 tablet_bootstrap.cc:654] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:13.610275 14456 log.cc:824] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:13.611057 14457 tablet_bootstrap.cc:654] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:13.612218 14457 log.cc:824] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:13.612752 14459 tablet_bootstrap.cc:492] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6: Bootstrap starting.
I20260620 07:10:13.614746 14456 tablet_bootstrap.cc:492] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b: No bootstrap required, opened a new log
I20260620 07:10:13.615052 14456 ts_tablet_manager.cc:1397] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 07:10:13.615492 14459 tablet_bootstrap.cc:654] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:13.616379 14459 log.cc:824] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:13.618897 14457 tablet_bootstrap.cc:492] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee: No bootstrap required, opened a new log
I20260620 07:10:13.619112 14459 tablet_bootstrap.cc:492] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6: No bootstrap required, opened a new log
I20260620 07:10:13.619138 14457 ts_tablet_manager.cc:1397] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260620 07:10:13.619351 14459 ts_tablet_manager.cc:1397] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260620 07:10:13.626026 14457 raft_consensus.cc:348] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } }
I20260620 07:10:13.626104 14456 raft_consensus.cc:348] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } }
I20260620 07:10:13.626401 14457 raft_consensus.cc:374] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:13.626494 14457 raft_consensus.cc:729] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f0514fc0604b3db4b4385605d73cee, State: Initialized, Role: FOLLOWER
I20260620 07:10:13.626564 14456 raft_consensus.cc:374] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:13.626688 14456 raft_consensus.cc:729] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd9e6b7b49f43abaa81b235d451959b, State: Initialized, Role: FOLLOWER
I20260620 07:10:13.627151 14457 consensus_queue.cc:260] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } }
I20260620 07:10:13.627276 14456 consensus_queue.cc:260] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } }
I20260620 07:10:13.628449 14293 heartbeater.cc:503] Master 127.6.53.126:40513 was elected leader, sending a full tablet report...
I20260620 07:10:13.628899 14456 ts_tablet_manager.cc:1428] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260620 07:10:13.629352 14459 raft_consensus.cc:348] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } }
I20260620 07:10:13.629901 14432 heartbeater.cc:503] Master 127.6.53.126:40513 was elected leader, sending a full tablet report...
I20260620 07:10:13.630337 14459 raft_consensus.cc:374] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:13.630486 14459 raft_consensus.cc:729] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 207fa9f0521d483aa525afb882a2b6b6, State: Initialized, Role: FOLLOWER
I20260620 07:10:13.630524 14457 ts_tablet_manager.cc:1428] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260620 07:10:13.631062 14459 consensus_queue.cc:260] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } }
I20260620 07:10:13.632696 14459 ts_tablet_manager.cc:1428] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260620 07:10:13.688232 14155 tablet.cc:2369] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:10:13.700695 14294 tablet.cc:2369] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:10:13.729159 14433 tablet.cc:2369] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:10:13.824857 14462 raft_consensus.cc:482] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:10:13.825187 14462 raft_consensus.cc:504] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } }
I20260620 07:10:13.827168 14462 leader_election.cc:288] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dd9e6b7b49f43abaa81b235d451959b (127.6.53.66:39999), 207fa9f0521d483aa525afb882a2b6b6 (127.6.53.65:43155)
I20260620 07:10:13.833698 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:13.827524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:34125 (local address 127.6.53.66:39999)
0620 07:10:13.827723 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:13.827729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:13.827750 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:10:13.827985 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:13.827991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:13.828062 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:13.828134 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:13.828705 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.829764 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.831735 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.832698 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.832841 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.832942 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:13.833308 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 07:10:13.833448 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:10:13.833742 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:13.827447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:39999 (local address 127.6.53.67:34125)
0620 07:10:13.827659 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:13.827680 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:13.827801 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:13.828285 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:13.828291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:13.828324 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:13.828544 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:13.828555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.829879 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.829884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:13.831590 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:13.831605 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.832856 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.832862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:13.833149 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.833188 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 07:10:13.833280 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 07:10:13.833458 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":5}
I20260620 07:10:13.835043 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c9ba8d0138408387b1ab2d583b266d" candidate_uuid: "09f0514fc0604b3db4b4385605d73cee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd9e6b7b49f43abaa81b235d451959b" is_pre_election: true
I20260620 07:10:13.835711 14243 raft_consensus.cc:2454] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f0514fc0604b3db4b4385605d73cee in term 0.
I20260620 07:10:13.836536 14316 leader_election.cc:302] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f0514fc0604b3db4b4385605d73cee, 1dd9e6b7b49f43abaa81b235d451959b; no voters:
I20260620 07:10:13.836750 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:13.827799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:43155 (local address 127.6.53.67:49941)
0620 07:10:13.828850 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:13.828864 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:13.829003 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:13.829334 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:13.829339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:13.829362 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:13.829667 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:13.829674 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.831012 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.831018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:13.833084 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:13.833098 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.835752 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.835759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:13.835961 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.836005 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 07:10:13.836107 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 07:10:13.836260 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":52,"thread_start_us":775,"threads_started":1}
I20260620 07:10:13.836704 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:13.828136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:49941 (local address 127.6.53.65:43155)
0620 07:10:13.828360 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:13.828368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:13.828956 (+ 588us) server_negotiation.cc:407] Connection header received
0620 07:10:13.829149 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:13.829152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:13.829217 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:13.829316 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:13.829812 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.830869 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.833223 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.834418 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.835498 (+ 1080us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.835613 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:13.836179 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 07:10:13.836364 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:10:13.836975 14462 raft_consensus.cc:2781] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:10:13.837116 14462 raft_consensus.cc:482] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:10:13.837210 14462 raft_consensus.cc:3037] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:13.837842 14109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c9ba8d0138408387b1ab2d583b266d" candidate_uuid: "09f0514fc0604b3db4b4385605d73cee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "207fa9f0521d483aa525afb882a2b6b6" is_pre_election: true
I20260620 07:10:13.838407 14109 raft_consensus.cc:2454] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f0514fc0604b3db4b4385605d73cee in term 0.
I20260620 07:10:13.840592 14462 raft_consensus.cc:504] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } }
I20260620 07:10:13.841240 14462 leader_election.cc:288] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [CANDIDATE]: Term 1 election: Requested vote from peers 1dd9e6b7b49f43abaa81b235d451959b (127.6.53.66:39999), 207fa9f0521d483aa525afb882a2b6b6 (127.6.53.65:43155)
I20260620 07:10:13.841704 14109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c9ba8d0138408387b1ab2d583b266d" candidate_uuid: "09f0514fc0604b3db4b4385605d73cee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "207fa9f0521d483aa525afb882a2b6b6"
I20260620 07:10:13.841683 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88c9ba8d0138408387b1ab2d583b266d" candidate_uuid: "09f0514fc0604b3db4b4385605d73cee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd9e6b7b49f43abaa81b235d451959b"
I20260620 07:10:13.841887 14109 raft_consensus.cc:3037] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:13.841912 14243 raft_consensus.cc:3037] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:13.845046 14243 raft_consensus.cc:2454] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f0514fc0604b3db4b4385605d73cee in term 1.
I20260620 07:10:13.845044 14109 raft_consensus.cc:2454] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f0514fc0604b3db4b4385605d73cee in term 1.
I20260620 07:10:13.845431 14316 leader_election.cc:302] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f0514fc0604b3db4b4385605d73cee, 1dd9e6b7b49f43abaa81b235d451959b; no voters:
I20260620 07:10:13.845680 14462 raft_consensus.cc:2781] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:10:13.846084 14462 raft_consensus.cc:686] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [term 1 LEADER]: Becoming Leader. State: Replica: 09f0514fc0604b3db4b4385605d73cee, State: Running, Role: LEADER
I20260620 07:10:13.846550 14462 consensus_queue.cc:237] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } }
I20260620 07:10:13.850644 13946 catalog_manager.cc:5495] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee reported cstate change: term changed from 0 to 1, leader changed from <none> to 09f0514fc0604b3db4b4385605d73cee (127.6.53.67). New cstate: current_term: 1 leader_uuid: "09f0514fc0604b3db4b4385605d73cee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } health_report { overall_health: UNKNOWN } } }
I20260620 07:10:13.876859 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:13.872112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34918 (local address 127.6.53.67:37775)
0620 07:10:13.872271 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:13.872276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:13.872292 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:10:13.872442 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:13.872446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:13.872517 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:13.872639 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:10:13.872932 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.873598 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.874883 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:13.875266 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:13.875354 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:13.875788 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:10:13.876288 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:10:13.876391 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:13.876495 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 07:10:13.876555 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:10:13.884418 14243 raft_consensus.cc:1261] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Refusing update from remote peer 09f0514fc0604b3db4b4385605d73cee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:10:13.884428 14109 raft_consensus.cc:1261] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Refusing update from remote peer 09f0514fc0604b3db4b4385605d73cee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:10:13.885277 14466 consensus_queue.cc:1035] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:13.885618 14462 consensus_queue.cc:1035] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [LEADER]: Connected to new peer: Peer: permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:13.896593 14471 mvcc.cc:205] Tried to move back new op lower bound from 7298823839260983296 to 7298823839118684160. Current Snapshot: MvccSnapshot[applied={T|T < 7298823839260983296}]
I20260620 07:10:13.907575 13947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:10:13.907843 13947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:10:13.911538 13947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3: Sending DeleteTablet for 3 replicas of tablet 88c9ba8d0138408387b1ab2d583b266d
I20260620 07:10:13.912665 14362 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c9ba8d0138408387b1ab2d583b266d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:13 UTC) from {username='oryx', principal='oryx/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:34912
I20260620 07:10:13.912709 14084 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c9ba8d0138408387b1ab2d583b266d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:13 UTC) from {username='oryx', principal='oryx/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:56568
I20260620 07:10:13.913695 14223 tablet_service.cc:1514] Processing DeleteTablet for tablet 88c9ba8d0138408387b1ab2d583b266d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:13 UTC) from {username='oryx', principal='oryx/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:42452
I20260620 07:10:13.914422 13946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:10:13.915043 13946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:10:13.923727 14361 tablet_service.cc:1467] Processing CreateTablet for tablet 39cc7a7b73d8438ea2b8fdd4d009b43a (DEFAULT_TABLE table=test-table [id=1d20dd5110ee4d2cb9384e15a6ccf356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:13.924068 14222 tablet_service.cc:1467] Processing CreateTablet for tablet 39cc7a7b73d8438ea2b8fdd4d009b43a (DEFAULT_TABLE table=test-table [id=1d20dd5110ee4d2cb9384e15a6ccf356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:13.924374 14361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39cc7a7b73d8438ea2b8fdd4d009b43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:13.924739 14222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39cc7a7b73d8438ea2b8fdd4d009b43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:13.931011 14083 tablet_service.cc:1467] Processing CreateTablet for tablet 39cc7a7b73d8438ea2b8fdd4d009b43a (DEFAULT_TABLE table=test-table [id=1d20dd5110ee4d2cb9384e15a6ccf356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:13.931532 14083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39cc7a7b73d8438ea2b8fdd4d009b43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:13.933852 14456 tablet_bootstrap.cc:492] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b: Bootstrap starting.
I20260620 07:10:13.935482 14456 tablet_bootstrap.cc:654] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:13.939585 14479 tablet_replica.cc:330] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee: stopping tablet replica
I20260620 07:10:13.939733 14480 tablet_replica.cc:330] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6: stopping tablet replica
I20260620 07:10:13.940133 14480 raft_consensus.cc:2229] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:10:13.940133 14479 raft_consensus.cc:2229] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:10:13.940696 14480 raft_consensus.cc:2258] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:13.940749 14479 raft_consensus.cc:2258] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:13.944376 14480 ts_tablet_manager.cc:1905] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:13.945458 14479 ts_tablet_manager.cc:1905] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:13.946303 14457 tablet_bootstrap.cc:492] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee: Bootstrap starting.
I20260620 07:10:13.947889 14457 tablet_bootstrap.cc:654] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:13.945441 14459 tablet_bootstrap.cc:492] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6: Bootstrap starting.
I20260620 07:10:13.953833 14481 tablet_replica.cc:330] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b: stopping tablet replica
I20260620 07:10:13.954222 14481 raft_consensus.cc:2229] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:10:13.954586 14459 tablet_bootstrap.cc:654] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:13.954728 14481 raft_consensus.cc:2258] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:13.957136 14457 tablet_bootstrap.cc:492] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee: No bootstrap required, opened a new log
I20260620 07:10:13.957321 14457 ts_tablet_manager.cc:1397] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260620 07:10:13.958171 14457 raft_consensus.cc:348] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:13.958513 14457 raft_consensus.cc:374] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:13.958640 14457 raft_consensus.cc:729] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f0514fc0604b3db4b4385605d73cee, State: Initialized, Role: FOLLOWER
I20260620 07:10:13.958930 14457 consensus_queue.cc:260] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:13.959561 14457 ts_tablet_manager.cc:1428] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:10:13.962416 14456 tablet_bootstrap.cc:492] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b: No bootstrap required, opened a new log
I20260620 07:10:13.962574 14456 ts_tablet_manager.cc:1397] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.016s
I20260620 07:10:13.966156 14456 raft_consensus.cc:348] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:13.966488 14456 raft_consensus.cc:374] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:13.966632 14456 raft_consensus.cc:729] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd9e6b7b49f43abaa81b235d451959b, State: Initialized, Role: FOLLOWER
I20260620 07:10:13.966977 14456 consensus_queue.cc:260] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:13.967509 14481 ts_tablet_manager.cc:1905] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:13.969678 14456 ts_tablet_manager.cc:1428] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260620 07:10:13.972854 14459 tablet_bootstrap.cc:492] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6: No bootstrap required, opened a new log
I20260620 07:10:13.973047 14459 ts_tablet_manager.cc:1397] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.000s
I20260620 07:10:13.974040 14459 raft_consensus.cc:348] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:13.974341 14459 raft_consensus.cc:374] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:13.974478 14459 raft_consensus.cc:729] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 207fa9f0521d483aa525afb882a2b6b6, State: Initialized, Role: FOLLOWER
I20260620 07:10:13.974874 14459 consensus_queue.cc:260] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:13.976837 14480 ts_tablet_manager.cc:1918] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:10:13.977044 14480 log.cc:1196] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/88c9ba8d0138408387b1ab2d583b266d
I20260620 07:10:13.977556 14459 ts_tablet_manager.cc:1428] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 07:10:13.977898 14479 ts_tablet_manager.cc:1918] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:10:13.978169 14479 log.cc:1196] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/88c9ba8d0138408387b1ab2d583b266d
I20260620 07:10:13.978775 14479 ts_tablet_manager.cc:1939] T 88c9ba8d0138408387b1ab2d583b266d P 09f0514fc0604b3db4b4385605d73cee: Deleting consensus metadata
I20260620 07:10:13.977645 14480 ts_tablet_manager.cc:1939] T 88c9ba8d0138408387b1ab2d583b266d P 207fa9f0521d483aa525afb882a2b6b6: Deleting consensus metadata
I20260620 07:10:13.980677 13932 catalog_manager.cc:4841] TS 09f0514fc0604b3db4b4385605d73cee (127.6.53.67:37775): tablet 88c9ba8d0138408387b1ab2d583b266d (table test-table [id=e0a5093bacc941578ee22c38e7156d10]) successfully deleted
I20260620 07:10:13.981034 13934 catalog_manager.cc:4841] TS 207fa9f0521d483aa525afb882a2b6b6 (127.6.53.65:43155): tablet 88c9ba8d0138408387b1ab2d583b266d (table test-table [id=e0a5093bacc941578ee22c38e7156d10]) successfully deleted
I20260620 07:10:13.983108 14481 ts_tablet_manager.cc:1918] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:10:13.983258 14481 log.cc:1196] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/88c9ba8d0138408387b1ab2d583b266d
I20260620 07:10:13.983866 14481 ts_tablet_manager.cc:1939] T 88c9ba8d0138408387b1ab2d583b266d P 1dd9e6b7b49f43abaa81b235d451959b: Deleting consensus metadata
I20260620 07:10:13.985455 13934 catalog_manager.cc:4841] TS 1dd9e6b7b49f43abaa81b235d451959b (127.6.53.66:39999): tablet 88c9ba8d0138408387b1ab2d583b266d (table test-table [id=e0a5093bacc941578ee22c38e7156d10]) successfully deleted
I20260620 07:10:14.239426 14474 raft_consensus.cc:482] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:10:14.239676 14474 raft_consensus.cc:504] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.240440 14474 leader_election.cc:288] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 207fa9f0521d483aa525afb882a2b6b6 (127.6.53.65:43155), 1dd9e6b7b49f43abaa81b235d451959b (127.6.53.66:39999)
I20260620 07:10:14.241060 14109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39cc7a7b73d8438ea2b8fdd4d009b43a" candidate_uuid: "09f0514fc0604b3db4b4385605d73cee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "207fa9f0521d483aa525afb882a2b6b6" is_pre_election: true
I20260620 07:10:14.241130 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39cc7a7b73d8438ea2b8fdd4d009b43a" candidate_uuid: "09f0514fc0604b3db4b4385605d73cee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd9e6b7b49f43abaa81b235d451959b" is_pre_election: true
I20260620 07:10:14.241289 14109 raft_consensus.cc:2454] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f0514fc0604b3db4b4385605d73cee in term 0.
I20260620 07:10:14.241421 14243 raft_consensus.cc:2454] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f0514fc0604b3db4b4385605d73cee in term 0.
I20260620 07:10:14.241815 14316 leader_election.cc:302] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f0514fc0604b3db4b4385605d73cee, 1dd9e6b7b49f43abaa81b235d451959b; no voters:
I20260620 07:10:14.242146 14474 raft_consensus.cc:2781] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:10:14.242333 14474 raft_consensus.cc:482] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:10:14.242475 14474 raft_consensus.cc:3037] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:14.245715 14474 raft_consensus.cc:504] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.246435 14474 leader_election.cc:288] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [CANDIDATE]: Term 1 election: Requested vote from peers 207fa9f0521d483aa525afb882a2b6b6 (127.6.53.65:43155), 1dd9e6b7b49f43abaa81b235d451959b (127.6.53.66:39999)
I20260620 07:10:14.246865 14109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39cc7a7b73d8438ea2b8fdd4d009b43a" candidate_uuid: "09f0514fc0604b3db4b4385605d73cee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "207fa9f0521d483aa525afb882a2b6b6"
I20260620 07:10:14.246938 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39cc7a7b73d8438ea2b8fdd4d009b43a" candidate_uuid: "09f0514fc0604b3db4b4385605d73cee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd9e6b7b49f43abaa81b235d451959b"
I20260620 07:10:14.247054 14109 raft_consensus.cc:3037] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:14.247161 14243 raft_consensus.cc:3037] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:14.249101 14243 raft_consensus.cc:2454] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f0514fc0604b3db4b4385605d73cee in term 1.
I20260620 07:10:14.249511 14316 leader_election.cc:302] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f0514fc0604b3db4b4385605d73cee, 1dd9e6b7b49f43abaa81b235d451959b; no voters:
I20260620 07:10:14.249799 14474 raft_consensus.cc:2781] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:10:14.249958 14474 raft_consensus.cc:686] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [term 1 LEADER]: Becoming Leader. State: Replica: 09f0514fc0604b3db4b4385605d73cee, State: Running, Role: LEADER
I20260620 07:10:14.250052 14109 raft_consensus.cc:2454] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f0514fc0604b3db4b4385605d73cee in term 1.
I20260620 07:10:14.250309 14474 consensus_queue.cc:237] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.253114 13947 catalog_manager.cc:5495] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee reported cstate change: term changed from 0 to 1, leader changed from <none> to 09f0514fc0604b3db4b4385605d73cee (127.6.53.67). New cstate: current_term: 1 leader_uuid: "09f0514fc0604b3db4b4385605d73cee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } health_report { overall_health: HEALTHY } } }
Jun 20 07:10:14 dist-test-slave-4n6h krb5kdc[13902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939411, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.126@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM
I20260620 07:10:14.353214 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:14.332357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:40513 (local address 127.0.0.1:32994)
0620 07:10:14.333184 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:14.333203 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:14.333342 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:14.333686 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:14.333691 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:14.334458 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:14.334630 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:14.334638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.335675 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.335680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:14.336652 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:14.336663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.337481 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.337487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:14.337668 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:14.345897 (+ 8229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:14.345914 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:14.348082 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:14.350826 (+ 2744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:14.350839 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:14.350847 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:14.351223 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:14.351579 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:14.351586 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:14.351589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:14.351720 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:14.352202 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:14.352210 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:14.352373 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 07:10:14.352535 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 07:10:14.352814 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":464,"threads_started":1}
I20260620 07:10:14.353251 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:14.332357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.6.53.126:40513)
0620 07:10:14.333168 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:14.333174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:14.333261 (+ 87us) server_negotiation.cc:407] Connection header received
0620 07:10:14.333475 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:14.333480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:14.333548 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:14.333652 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:14.334758 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.335551 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.336799 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.337322 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.337464 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:14.348266 (+ 10802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:14.348270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:14.348273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:14.348299 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:14.350705 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:14.351333 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:14.351340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:14.351345 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:14.351475 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:14.351828 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:14.351833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:14.351836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:14.352072 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:14.352215 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:14.352605 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 07:10:14.352967 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":167,"server-negotiator.queue_time_us":647,"thread_start_us":456,"threads_started":1}
I20260620 07:10:14.357861 13947 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:32994:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 07:10:14.359189 13947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:10:14.366017 14223 tablet_service.cc:1467] Processing CreateTablet for tablet dddbdc5481614cfb90c338c386d49c10 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ebd4657bace841b4bd0d777957273152]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:10:14.366146 14083 tablet_service.cc:1467] Processing CreateTablet for tablet dddbdc5481614cfb90c338c386d49c10 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ebd4657bace841b4bd0d777957273152]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:10:14.366147 14361 tablet_service.cc:1467] Processing CreateTablet for tablet dddbdc5481614cfb90c338c386d49c10 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ebd4657bace841b4bd0d777957273152]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:10:14.366590 14223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dddbdc5481614cfb90c338c386d49c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:14.366595 14083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dddbdc5481614cfb90c338c386d49c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:14.366701 14361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dddbdc5481614cfb90c338c386d49c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:14.372112 14457 tablet_bootstrap.cc:492] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee: Bootstrap starting.
I20260620 07:10:14.373437 14456 tablet_bootstrap.cc:492] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b: Bootstrap starting.
I20260620 07:10:14.373483 14457 tablet_bootstrap.cc:654] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:14.375171 14456 tablet_bootstrap.cc:654] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:14.375543 14457 tablet_bootstrap.cc:492] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee: No bootstrap required, opened a new log
I20260620 07:10:14.375689 14457 ts_tablet_manager.cc:1397] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260620 07:10:14.376127 14459 tablet_bootstrap.cc:492] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6: Bootstrap starting.
I20260620 07:10:14.376428 14457 raft_consensus.cc:348] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.376890 14457 raft_consensus.cc:374] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:14.377012 14457 raft_consensus.cc:729] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f0514fc0604b3db4b4385605d73cee, State: Initialized, Role: FOLLOWER
I20260620 07:10:14.377292 14457 consensus_queue.cc:260] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.377691 14459 tablet_bootstrap.cc:654] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:14.377827 14457 ts_tablet_manager.cc:1428] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 07:10:14.377739 14474 tablet_replica.cc:439] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } }
I20260620 07:10:14.378008 14474 tablet_replica.cc:442] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:10:14.379958 14459 tablet_bootstrap.cc:492] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6: No bootstrap required, opened a new log
I20260620 07:10:14.380076 14456 tablet_bootstrap.cc:492] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b: No bootstrap required, opened a new log
I20260620 07:10:14.380126 14459 ts_tablet_manager.cc:1397] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 07:10:14.380231 14456 ts_tablet_manager.cc:1397] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260620 07:10:14.380805 14459 raft_consensus.cc:348] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.381037 14459 raft_consensus.cc:374] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:14.380985 14456 raft_consensus.cc:348] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.381117 14459 raft_consensus.cc:729] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 207fa9f0521d483aa525afb882a2b6b6, State: Initialized, Role: FOLLOWER
I20260620 07:10:14.381258 14456 raft_consensus.cc:374] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:14.381376 14456 raft_consensus.cc:729] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd9e6b7b49f43abaa81b235d451959b, State: Initialized, Role: FOLLOWER
I20260620 07:10:14.381376 14459 consensus_queue.cc:260] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.381629 14456 consensus_queue.cc:260] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.381958 14459 ts_tablet_manager.cc:1428] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:10:14.381835 14464 tablet_replica.cc:439] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } }
I20260620 07:10:14.382155 14464 tablet_replica.cc:442] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:10:14.382189 14456 ts_tablet_manager.cc:1428] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 07:10:14.382088 14463 tablet_replica.cc:439] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } }
I20260620 07:10:14.382419 14463 tablet_replica.cc:442] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:10:14.517031 14463 raft_consensus.cc:482] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:10:14.517218 14463 raft_consensus.cc:504] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.518671 14463 leader_election.cc:288] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 207fa9f0521d483aa525afb882a2b6b6 (127.6.53.65:43155), 09f0514fc0604b3db4b4385605d73cee (127.6.53.67:37775)
I20260620 07:10:14.537590 14498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:14.518988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:46797 (local address 127.6.53.65:43155)
0620 07:10:14.531362 (+ 12374us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:14.531391 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:14.531662 (+ 271us) server_negotiation.cc:407] Connection header received
0620 07:10:14.531899 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:14.531903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:14.531959 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:14.532070 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:14.532732 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.533541 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.535447 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.536657 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.536749 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:14.536819 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:14.537188 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 07:10:14.537339 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12208,"thread_start_us":12099,"threads_started":1}
I20260620 07:10:14.537671 14497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:14.518931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:43155 (local address 127.6.53.66:46797)
0620 07:10:14.531545 (+ 12614us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:14.531562 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:14.531739 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:14.532086 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:14.532090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:14.532115 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:14.532497 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:14.532510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.533685 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.533690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:14.535294 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:14.535305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.536799 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.536805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:14.536968 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:14.537004 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:10:14.537144 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 07:10:14.537290 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12498,"thread_start_us":13191,"threads_started":1}
I20260620 07:10:14.538422 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:14.519136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:33851 (local address 127.6.53.67:37775)
0620 07:10:14.531895 (+ 12759us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:14.531904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:14.532455 (+ 551us) server_negotiation.cc:407] Connection header received
0620 07:10:14.532562 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:14.532570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:14.532662 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:14.532805 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:14.533358 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.534335 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.536384 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.537376 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.537500 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:14.537594 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:14.537998 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 07:10:14.538124 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12606,"thread_start_us":12764,"threads_started":1}
I20260620 07:10:14.538455 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:14.518967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:37775 (local address 127.6.53.66:33851)
0620 07:10:14.532345 (+ 13378us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:14.532358 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:14.532461 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:14.532815 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:14.532819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:14.532842 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:14.533170 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:14.533177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.534471 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.534474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:14.536224 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:14.536238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.537578 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.537585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:14.537771 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:14.537806 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:10:14.537929 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 07:10:14.538053 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13257,"mutex_wait_us":1,"thread_start_us":12349,"threads_started":1}
I20260620 07:10:14.538393 14109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dddbdc5481614cfb90c338c386d49c10" candidate_uuid: "1dd9e6b7b49f43abaa81b235d451959b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "207fa9f0521d483aa525afb882a2b6b6" is_pre_election: true
I20260620 07:10:14.538661 14109 raft_consensus.cc:2454] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dd9e6b7b49f43abaa81b235d451959b in term 0.
I20260620 07:10:14.539112 14177 leader_election.cc:302] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd9e6b7b49f43abaa81b235d451959b, 207fa9f0521d483aa525afb882a2b6b6; no voters:
I20260620 07:10:14.539278 14382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dddbdc5481614cfb90c338c386d49c10" candidate_uuid: "1dd9e6b7b49f43abaa81b235d451959b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f0514fc0604b3db4b4385605d73cee" is_pre_election: true
I20260620 07:10:14.539494 14463 raft_consensus.cc:2781] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:10:14.539604 14463 raft_consensus.cc:482] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:10:14.539702 14463 raft_consensus.cc:3037] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:14.539685 14382 raft_consensus.cc:2454] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dd9e6b7b49f43abaa81b235d451959b in term 0.
I20260620 07:10:14.541783 14463 raft_consensus.cc:504] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.542380 14463 leader_election.cc:288] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [CANDIDATE]: Term 1 election: Requested vote from peers 207fa9f0521d483aa525afb882a2b6b6 (127.6.53.65:43155), 09f0514fc0604b3db4b4385605d73cee (127.6.53.67:37775)
I20260620 07:10:14.542742 14109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dddbdc5481614cfb90c338c386d49c10" candidate_uuid: "1dd9e6b7b49f43abaa81b235d451959b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "207fa9f0521d483aa525afb882a2b6b6"
I20260620 07:10:14.542884 14382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dddbdc5481614cfb90c338c386d49c10" candidate_uuid: "1dd9e6b7b49f43abaa81b235d451959b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09f0514fc0604b3db4b4385605d73cee"
I20260620 07:10:14.542954 14109 raft_consensus.cc:3037] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:14.543084 14382 raft_consensus.cc:3037] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:14.544857 14109 raft_consensus.cc:2454] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dd9e6b7b49f43abaa81b235d451959b in term 1.
I20260620 07:10:14.545173 14382 raft_consensus.cc:2454] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dd9e6b7b49f43abaa81b235d451959b in term 1.
I20260620 07:10:14.545188 14177 leader_election.cc:302] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dd9e6b7b49f43abaa81b235d451959b, 207fa9f0521d483aa525afb882a2b6b6; no voters:
I20260620 07:10:14.545943 14463 raft_consensus.cc:2781] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:10:14.546459 14463 raft_consensus.cc:686] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [term 1 LEADER]: Becoming Leader. State: Replica: 1dd9e6b7b49f43abaa81b235d451959b, State: Running, Role: LEADER
I20260620 07:10:14.546864 14463 consensus_queue.cc:237] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.548420 14500 tablet_replica.cc:439] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b: TxnStatusTablet state changed. Reason: New leader 1dd9e6b7b49f43abaa81b235d451959b. Latest consensus state: current_term: 1 leader_uuid: "1dd9e6b7b49f43abaa81b235d451959b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } }
I20260620 07:10:14.548733 14500 tablet_replica.cc:442] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:10:14.549428 14502 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 07:10:14.549636 14502 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 07:10:14.550122 13946 catalog_manager.cc:5495] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dd9e6b7b49f43abaa81b235d451959b (127.6.53.66). New cstate: current_term: 1 leader_uuid: "1dd9e6b7b49f43abaa81b235d451959b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } health_report { overall_health: UNKNOWN } } }
I20260620 07:10:14.572067 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:14.565896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:39999 (local address 127.0.0.1:42468)
0620 07:10:14.566110 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:14.566135 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:14.566315 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:14.566900 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:14.566904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:14.566919 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:10:14.567233 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:14.567252 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.568448 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.568475 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:14.569551 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:14.569563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.570373 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.570378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:14.570560 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:14.570610 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:10:14.571510 (+ 900us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:10:14.571530 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 07:10:14.571615 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:10:14.571760 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 07:10:14.572108 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:14.565896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42468 (local address 127.6.53.66:39999)
0620 07:10:14.566556 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:14.566561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:14.566578 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:10:14.566642 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:14.566645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:14.566728 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:14.566881 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:10:14.567486 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.568314 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.569701 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.570185 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.570362 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:14.570837 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:10:14.571394 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:10:14.571550 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:14.571652 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 07:10:14.571731 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":352,"threads_started":1}
I20260620 07:10:14.576484 14109 raft_consensus.cc:1261] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Refusing update from remote peer 1dd9e6b7b49f43abaa81b235d451959b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:10:14.576696 14382 raft_consensus.cc:1261] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee [term 1 FOLLOWER]: Refusing update from remote peer 1dd9e6b7b49f43abaa81b235d451959b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:10:14.577131 14463 consensus_queue.cc:1035] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [LEADER]: Connected to new peer: Peer: permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:14.577512 14500 consensus_queue.cc:1035] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b [LEADER]: Connected to new peer: Peer: permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:14.583043 14474 tablet_replica.cc:439] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee: TxnStatusTablet state changed. Reason: New leader 1dd9e6b7b49f43abaa81b235d451959b. Latest consensus state: current_term: 1 leader_uuid: "1dd9e6b7b49f43abaa81b235d451959b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } }
I20260620 07:10:14.583330 14474 tablet_replica.cc:442] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:10:14.586876 14506 mvcc.cc:205] Tried to move back new op lower bound from 7298823842098405376 to 7298823841986822144. Current Snapshot: MvccSnapshot[applied={T|T < 7298823842098405376}]
I20260620 07:10:14.587812 14508 mvcc.cc:205] Tried to move back new op lower bound from 7298823842098405376 to 7298823841986822144. Current Snapshot: MvccSnapshot[applied={T|T < 7298823842098405376}]
I20260620 07:10:14.588646 14464 tablet_replica.cc:439] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6: TxnStatusTablet state changed. Reason: New leader 1dd9e6b7b49f43abaa81b235d451959b. Latest consensus state: current_term: 1 leader_uuid: "1dd9e6b7b49f43abaa81b235d451959b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } }
I20260620 07:10:14.589003 14464 tablet_replica.cc:442] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:10:14.595444 14474 tablet_replica.cc:439] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1dd9e6b7b49f43abaa81b235d451959b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } }
I20260620 07:10:14.595722 14474 tablet_replica.cc:442] T dddbdc5481614cfb90c338c386d49c10 P 09f0514fc0604b3db4b4385605d73cee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:10:14.602173 14464 tablet_replica.cc:439] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1dd9e6b7b49f43abaa81b235d451959b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } }
I20260620 07:10:14.603358 14464 tablet_replica.cc:442] T dddbdc5481614cfb90c338c386d49c10 P 207fa9f0521d483aa525afb882a2b6b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:10:14.597745 14463 tablet_replica.cc:439] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1dd9e6b7b49f43abaa81b235d451959b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } }
I20260620 07:10:14.604398 14463 tablet_replica.cc:442] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:10:14.604676 14463 tablet_replica.cc:439] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1dd9e6b7b49f43abaa81b235d451959b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } } }
I20260620 07:10:14.606494 14463 tablet_replica.cc:442] T dddbdc5481614cfb90c338c386d49c10 P 1dd9e6b7b49f43abaa81b235d451959b: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:10:14.629969 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:14.624550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:34959 (local address 127.6.53.66:39999)
0620 07:10:14.624750 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:14.624759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:14.625154 (+ 395us) server_negotiation.cc:407] Connection header received
0620 07:10:14.625383 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:14.625387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:14.625453 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:14.625565 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:10:14.626111 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.626897 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.628244 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.628658 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.628795 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:14.629211 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:10:14.629317 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:10:14.629412 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:14.629571 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 07:10:14.629647 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:10:14.630072 14519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:14.624409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:39999 (local address 127.6.53.67:34959)
0620 07:10:14.625046 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:14.625061 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:14.625233 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:14.625585 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:14.625589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:14.625601 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:10:14.625940 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:14.625949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.627032 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.627039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:14.628090 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:14.628100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.628830 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.628838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:14.628976 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:14.629027 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:10:14.629413 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:10:14.629427 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 07:10:14.629547 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 07:10:14.629677 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":331,"threads_started":1}
I20260620 07:10:14.639436 14243 raft_consensus.cc:1261] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Refusing update from remote peer 09f0514fc0604b3db4b4385605d73cee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:10:14.639457 14107 raft_consensus.cc:1261] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Refusing update from remote peer 09f0514fc0604b3db4b4385605d73cee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:10:14.640021 14474 consensus_queue.cc:1035] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [LEADER]: Connected to new peer: Peer: permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:14.640300 14466 consensus_queue.cc:1035] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:14.677757 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:14.670948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:47473 (local address 127.6.53.67:37775)
0620 07:10:14.671120 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:14.671126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:14.671361 (+ 235us) server_negotiation.cc:407] Connection header received
0620 07:10:14.671563 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:14.671569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:14.671659 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:14.671762 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:10:14.672435 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.673533 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.675582 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.676298 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.676424 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:14.676997 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:10:14.677124 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:10:14.677216 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:14.677362 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 07:10:14.677447 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:10:14.677814 14523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:14.670623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:37775 (local address 127.6.53.66:47473)
0620 07:10:14.671247 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:14.671270 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:14.671425 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:14.671793 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:14.671797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:14.671809 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:10:14.672250 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:14.672263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.673674 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.673678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:14.675393 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:14.675405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.676478 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.676482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:14.676696 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:14.676797 (+ 101us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:10:14.677235 (+ 438us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:10:14.677250 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 07:10:14.677312 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 07:10:14.677414 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":506,"threads_started":1}
I20260620 07:10:14.709285 13947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:10:14.709519 13947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:10:14.712510 13947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3: Sending DeleteTablet for 3 replicas of tablet 39cc7a7b73d8438ea2b8fdd4d009b43a
I20260620 07:10:14.713222 14083 tablet_service.cc:1514] Processing DeleteTablet for tablet 39cc7a7b73d8438ea2b8fdd4d009b43a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:14 UTC) from {username='oryx', principal='oryx/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:56568
I20260620 07:10:14.713312 14223 tablet_service.cc:1514] Processing DeleteTablet for tablet 39cc7a7b73d8438ea2b8fdd4d009b43a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:14 UTC) from {username='oryx', principal='oryx/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:42452
I20260620 07:10:14.718662 14361 tablet_service.cc:1514] Processing DeleteTablet for tablet 39cc7a7b73d8438ea2b8fdd4d009b43a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:14 UTC) from {username='oryx', principal='oryx/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:34912
I20260620 07:10:14.725104 14526 tablet_replica.cc:330] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b: stopping tablet replica
I20260620 07:10:14.725410 14526 raft_consensus.cc:2229] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:10:14.725768 14526 raft_consensus.cc:2258] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:14.726389 14525 tablet_replica.cc:330] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6: stopping tablet replica
I20260620 07:10:14.726612 14525 raft_consensus.cc:2229] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:10:14.726876 14525 raft_consensus.cc:2258] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:14.727758 14526 ts_tablet_manager.cc:1905] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:14.732057 14525 ts_tablet_manager.cc:1905] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:14.738675 14527 tablet_replica.cc:330] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee: stopping tablet replica
I20260620 07:10:14.738888 14527 raft_consensus.cc:2229] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:10:14.739168 14525 ts_tablet_manager.cc:1918] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:10:14.739297 14525 log.cc:1196] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/39cc7a7b73d8438ea2b8fdd4d009b43a
I20260620 07:10:14.739318 14527 raft_consensus.cc:2258] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:14.739774 14525 ts_tablet_manager.cc:1939] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 207fa9f0521d483aa525afb882a2b6b6: Deleting consensus metadata
I20260620 07:10:14.740227 14527 ts_tablet_manager.cc:1905] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:14.740938 13934 catalog_manager.cc:4841] TS 207fa9f0521d483aa525afb882a2b6b6 (127.6.53.65:43155): tablet 39cc7a7b73d8438ea2b8fdd4d009b43a (table test-table [id=1d20dd5110ee4d2cb9384e15a6ccf356]) successfully deleted
I20260620 07:10:14.744828 14526 ts_tablet_manager.cc:1918] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:10:14.744974 14526 log.cc:1196] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/39cc7a7b73d8438ea2b8fdd4d009b43a
I20260620 07:10:14.745252 14527 ts_tablet_manager.cc:1918] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:10:14.745393 14527 log.cc:1196] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/39cc7a7b73d8438ea2b8fdd4d009b43a
I20260620 07:10:14.745469 14526 ts_tablet_manager.cc:1939] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 1dd9e6b7b49f43abaa81b235d451959b: Deleting consensus metadata
I20260620 07:10:14.745805 14527 ts_tablet_manager.cc:1939] T 39cc7a7b73d8438ea2b8fdd4d009b43a P 09f0514fc0604b3db4b4385605d73cee: Deleting consensus metadata
I20260620 07:10:14.746824 13932 catalog_manager.cc:4841] TS 09f0514fc0604b3db4b4385605d73cee (127.6.53.67:37775): tablet 39cc7a7b73d8438ea2b8fdd4d009b43a (table test-table [id=1d20dd5110ee4d2cb9384e15a6ccf356]) successfully deleted
I20260620 07:10:14.747103 13934 catalog_manager.cc:4841] TS 1dd9e6b7b49f43abaa81b235d451959b (127.6.53.66:39999): tablet 39cc7a7b73d8438ea2b8fdd4d009b43a (table test-table [id=1d20dd5110ee4d2cb9384e15a6ccf356]) successfully deleted
Jun 20 07:10:14 dist-test-slave-4n6h krb5kdc[13902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:10:14 dist-test-slave-4n6h krb5kdc[13902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:14 dist-test-slave-4n6h krb5kdc[13902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 07:10:14.785983 14491 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:14.774639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.6.53.126:40513)
0620 07:10:14.774906 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:14.774914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:14.775376 (+ 462us) server_negotiation.cc:407] Connection header received
0620 07:10:14.775524 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:14.775528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:14.775590 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:14.775730 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:14.777420 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.778151 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.780012 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.780421 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.780514 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:14.785730 (+ 5216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32996: BlockingRecv error: recv got EOF from 127.0.0.1:32996 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 07:10:14 dist-test-slave-4n6h krb5kdc[13902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM
I20260620 07:10:14.804229 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:14.788754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33006 (local address 127.6.53.126:40513)
0620 07:10:14.788962 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:14.788968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:14.789247 (+ 279us) server_negotiation.cc:407] Connection header received
0620 07:10:14.789397 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:14.789401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:14.789454 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:14.789531 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:14.791002 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.791813 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.793598 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:14.794059 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:14.794165 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:14.798387 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:14.798393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:14.798399 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:14.798439 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:14.801633 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:14.802241 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:14.802265 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:14.802269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:14.802418 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:14.802856 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:14.802861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:14.802864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:14.803176 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:14.803424 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:14.803615 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 07:10:14.803837 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:10:14.807778 13947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:10:14.808429 13947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:10:14.815241 14083 tablet_service.cc:1467] Processing CreateTablet for tablet e22976aa256e4cdcaa3274a597475c9f (DEFAULT_TABLE table=test-table [id=abdf94402e4541c4846256eafce0c9d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:14.815641 14361 tablet_service.cc:1467] Processing CreateTablet for tablet e22976aa256e4cdcaa3274a597475c9f (DEFAULT_TABLE table=test-table [id=abdf94402e4541c4846256eafce0c9d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:14.815714 14083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e22976aa256e4cdcaa3274a597475c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:14.816041 14361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e22976aa256e4cdcaa3274a597475c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:14.819133 14223 tablet_service.cc:1467] Processing CreateTablet for tablet e22976aa256e4cdcaa3274a597475c9f (DEFAULT_TABLE table=test-table [id=abdf94402e4541c4846256eafce0c9d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:14.819541 14223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e22976aa256e4cdcaa3274a597475c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:14.822014 14459 tablet_bootstrap.cc:492] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6: Bootstrap starting.
I20260620 07:10:14.823540 14459 tablet_bootstrap.cc:654] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:14.825947 14456 tablet_bootstrap.cc:492] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b: Bootstrap starting.
I20260620 07:10:14.827634 14456 tablet_bootstrap.cc:654] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:14.827833 14459 tablet_bootstrap.cc:492] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6: No bootstrap required, opened a new log
I20260620 07:10:14.827958 14459 ts_tablet_manager.cc:1397] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 07:10:14.828617 14459 raft_consensus.cc:348] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.828868 14459 raft_consensus.cc:374] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:14.828953 14459 raft_consensus.cc:729] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 207fa9f0521d483aa525afb882a2b6b6, State: Initialized, Role: FOLLOWER
I20260620 07:10:14.829177 14459 consensus_queue.cc:260] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.831765 14459 ts_tablet_manager.cc:1428] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 07:10:14.835726 14456 tablet_bootstrap.cc:492] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b: No bootstrap required, opened a new log
I20260620 07:10:14.835878 14456 ts_tablet_manager.cc:1397] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260620 07:10:14.836077 14457 tablet_bootstrap.cc:492] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee: Bootstrap starting.
I20260620 07:10:14.836620 14456 raft_consensus.cc:348] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.836853 14456 raft_consensus.cc:374] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:14.836944 14456 raft_consensus.cc:729] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd9e6b7b49f43abaa81b235d451959b, State: Initialized, Role: FOLLOWER
I20260620 07:10:14.837183 14456 consensus_queue.cc:260] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.837636 14457 tablet_bootstrap.cc:654] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:14.840152 14456 ts_tablet_manager.cc:1428] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 07:10:14.843323 14457 tablet_bootstrap.cc:492] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee: No bootstrap required, opened a new log
I20260620 07:10:14.843480 14457 ts_tablet_manager.cc:1397] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260620 07:10:14.844193 14457 raft_consensus.cc:348] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.844424 14457 raft_consensus.cc:374] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:14.844509 14457 raft_consensus.cc:729] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f0514fc0604b3db4b4385605d73cee, State: Initialized, Role: FOLLOWER
I20260620 07:10:14.844744 14457 consensus_queue.cc:260] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:14.845350 14457 ts_tablet_manager.cc:1428] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 07:10:15.022570 14474 raft_consensus.cc:482] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:10:15.022743 14474 raft_consensus.cc:504] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:15.023368 14474 leader_election.cc:288] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 207fa9f0521d483aa525afb882a2b6b6 (127.6.53.65:43155), 1dd9e6b7b49f43abaa81b235d451959b (127.6.53.66:39999)
I20260620 07:10:15.024044 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e22976aa256e4cdcaa3274a597475c9f" candidate_uuid: "09f0514fc0604b3db4b4385605d73cee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd9e6b7b49f43abaa81b235d451959b" is_pre_election: true
I20260620 07:10:15.024025 14107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e22976aa256e4cdcaa3274a597475c9f" candidate_uuid: "09f0514fc0604b3db4b4385605d73cee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "207fa9f0521d483aa525afb882a2b6b6" is_pre_election: true
I20260620 07:10:15.024310 14243 raft_consensus.cc:2454] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f0514fc0604b3db4b4385605d73cee in term 0.
I20260620 07:10:15.024327 14107 raft_consensus.cc:2454] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f0514fc0604b3db4b4385605d73cee in term 0.
I20260620 07:10:15.024770 14316 leader_election.cc:302] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f0514fc0604b3db4b4385605d73cee, 207fa9f0521d483aa525afb882a2b6b6; no voters:
I20260620 07:10:15.025012 14474 raft_consensus.cc:2781] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:10:15.025125 14474 raft_consensus.cc:482] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:10:15.025257 14474 raft_consensus.cc:3037] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:15.027135 14474 raft_consensus.cc:504] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:15.027642 14474 leader_election.cc:288] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [CANDIDATE]: Term 1 election: Requested vote from peers 207fa9f0521d483aa525afb882a2b6b6 (127.6.53.65:43155), 1dd9e6b7b49f43abaa81b235d451959b (127.6.53.66:39999)
I20260620 07:10:15.028102 14107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e22976aa256e4cdcaa3274a597475c9f" candidate_uuid: "09f0514fc0604b3db4b4385605d73cee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "207fa9f0521d483aa525afb882a2b6b6"
I20260620 07:10:15.028180 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e22976aa256e4cdcaa3274a597475c9f" candidate_uuid: "09f0514fc0604b3db4b4385605d73cee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd9e6b7b49f43abaa81b235d451959b"
I20260620 07:10:15.028327 14107 raft_consensus.cc:3037] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:15.028364 14243 raft_consensus.cc:3037] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:15.031224 14107 raft_consensus.cc:2454] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f0514fc0604b3db4b4385605d73cee in term 1.
I20260620 07:10:15.031247 14243 raft_consensus.cc:2454] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f0514fc0604b3db4b4385605d73cee in term 1.
I20260620 07:10:15.031615 14316 leader_election.cc:302] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f0514fc0604b3db4b4385605d73cee, 207fa9f0521d483aa525afb882a2b6b6; no voters:
I20260620 07:10:15.031919 14474 raft_consensus.cc:2781] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:10:15.032073 14474 raft_consensus.cc:686] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [term 1 LEADER]: Becoming Leader. State: Replica: 09f0514fc0604b3db4b4385605d73cee, State: Running, Role: LEADER
I20260620 07:10:15.032352 14474 consensus_queue.cc:237] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:15.034776 13947 catalog_manager.cc:5495] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee reported cstate change: term changed from 0 to 1, leader changed from <none> to 09f0514fc0604b3db4b4385605d73cee (127.6.53.67). New cstate: current_term: 1 leader_uuid: "09f0514fc0604b3db4b4385605d73cee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } health_report { overall_health: HEALTHY } } }
I20260620 07:10:15.071169 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:15.066768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.6.53.67:37775)
0620 07:10:15.066926 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:15.066930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:15.066947 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:10:15.067032 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:15.067036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:15.067099 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:15.067172 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:10:15.067468 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:15.068270 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:15.069564 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:15.069967 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:15.070058 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:15.070492 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:10:15.070584 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:10:15.070681 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:15.070825 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 07:10:15.070906 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:10:15.073709 14243 raft_consensus.cc:1261] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Refusing update from remote peer 09f0514fc0604b3db4b4385605d73cee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:10:15.073769 14107 raft_consensus.cc:1261] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Refusing update from remote peer 09f0514fc0604b3db4b4385605d73cee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:10:15.074429 14474 consensus_queue.cc:1035] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:15.074666 14466 consensus_queue.cc:1035] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [LEADER]: Connected to new peer: Peer: permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:15.085417 13946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:10:15.085625 13946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:10:15.088269 13946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3: Sending DeleteTablet for 3 replicas of tablet e22976aa256e4cdcaa3274a597475c9f
I20260620 07:10:15.089044 14223 tablet_service.cc:1514] Processing DeleteTablet for tablet e22976aa256e4cdcaa3274a597475c9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:15 UTC) from {username='oryx', principal='oryx/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:42452
I20260620 07:10:15.089066 14083 tablet_service.cc:1514] Processing DeleteTablet for tablet e22976aa256e4cdcaa3274a597475c9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:15 UTC) from {username='oryx', principal='oryx/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:56568
I20260620 07:10:15.089304 14525 tablet_replica.cc:330] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6: stopping tablet replica
I20260620 07:10:15.089325 14526 tablet_replica.cc:330] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b: stopping tablet replica
I20260620 07:10:15.089502 14525 raft_consensus.cc:2229] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:10:15.089535 14526 raft_consensus.cc:2229] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:10:15.089730 14525 raft_consensus.cc:2258] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:15.089749 14526 raft_consensus.cc:2258] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:15.090152 14361 tablet_service.cc:1514] Processing DeleteTablet for tablet e22976aa256e4cdcaa3274a597475c9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:15 UTC) from {username='oryx', principal='oryx/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:34912
I20260620 07:10:15.090070 13946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 07:10:15.090464 14527 tablet_replica.cc:330] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee: stopping tablet replica
I20260620 07:10:15.090613 14525 ts_tablet_manager.cc:1905] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:15.090639 14527 raft_consensus.cc:2229] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [term 1 LEADER]: Raft consensus shutting down.
W20260620 07:10:15.090665 13946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:10:15.090905 14527 raft_consensus.cc:2258] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:15.091871 14526 ts_tablet_manager.cc:1905] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:15.094837 14527 ts_tablet_manager.cc:1905] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:15.096501 14525 ts_tablet_manager.cc:1918] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:10:15.096652 14525 log.cc:1196] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/e22976aa256e4cdcaa3274a597475c9f
I20260620 07:10:15.097143 14525 ts_tablet_manager.cc:1939] T e22976aa256e4cdcaa3274a597475c9f P 207fa9f0521d483aa525afb882a2b6b6: Deleting consensus metadata
I20260620 07:10:15.097198 14083 tablet_service.cc:1467] Processing CreateTablet for tablet 7d367df2bdf24520a1d8ea2478430ece (DEFAULT_TABLE table=test-table [id=25dbde7ce3434035b4e4c40a58536633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:15.097426 14223 tablet_service.cc:1467] Processing CreateTablet for tablet 7d367df2bdf24520a1d8ea2478430ece (DEFAULT_TABLE table=test-table [id=25dbde7ce3434035b4e4c40a58536633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:15.097661 14083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d367df2bdf24520a1d8ea2478430ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:15.097862 14223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d367df2bdf24520a1d8ea2478430ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:15.101454 14459 tablet_bootstrap.cc:492] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6: Bootstrap starting.
I20260620 07:10:15.102205 14456 tablet_bootstrap.cc:492] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b: Bootstrap starting.
I20260620 07:10:15.102989 14459 tablet_bootstrap.cc:654] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:15.104086 14456 tablet_bootstrap.cc:654] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:15.105098 14459 tablet_bootstrap.cc:492] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6: No bootstrap required, opened a new log
I20260620 07:10:15.105295 14459 ts_tablet_manager.cc:1397] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 07:10:15.106101 14456 tablet_bootstrap.cc:492] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b: No bootstrap required, opened a new log
I20260620 07:10:15.106067 14459 raft_consensus.cc:348] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:15.106277 14456 ts_tablet_manager.cc:1397] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 07:10:15.106336 14459 raft_consensus.cc:374] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:15.106416 14459 raft_consensus.cc:729] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 207fa9f0521d483aa525afb882a2b6b6, State: Initialized, Role: FOLLOWER
I20260620 07:10:15.106662 14459 consensus_queue.cc:260] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:15.107038 14456 raft_consensus.cc:348] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:15.107265 14456 raft_consensus.cc:374] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:15.107352 14456 raft_consensus.cc:729] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd9e6b7b49f43abaa81b235d451959b, State: Initialized, Role: FOLLOWER
I20260620 07:10:15.107580 14527 ts_tablet_manager.cc:1918] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:10:15.107728 14527 log.cc:1196] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/e22976aa256e4cdcaa3274a597475c9f
I20260620 07:10:15.107604 14456 consensus_queue.cc:260] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:15.107736 14459 ts_tablet_manager.cc:1428] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:10:15.108098 14527 ts_tablet_manager.cc:1939] T e22976aa256e4cdcaa3274a597475c9f P 09f0514fc0604b3db4b4385605d73cee: Deleting consensus metadata
I20260620 07:10:15.108088 14456 ts_tablet_manager.cc:1428] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 07:10:15.108718 13934 catalog_manager.cc:4841] TS 207fa9f0521d483aa525afb882a2b6b6 (127.6.53.65:43155): tablet e22976aa256e4cdcaa3274a597475c9f (table test-table [id=abdf94402e4541c4846256eafce0c9d8]) successfully deleted
I20260620 07:10:15.109202 13932 catalog_manager.cc:4841] TS 09f0514fc0604b3db4b4385605d73cee (127.6.53.67:37775): tablet e22976aa256e4cdcaa3274a597475c9f (table test-table [id=abdf94402e4541c4846256eafce0c9d8]) successfully deleted
I20260620 07:10:15.109582 14361 tablet_service.cc:1467] Processing CreateTablet for tablet 7d367df2bdf24520a1d8ea2478430ece (DEFAULT_TABLE table=test-table [id=25dbde7ce3434035b4e4c40a58536633]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:15.109966 14361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d367df2bdf24520a1d8ea2478430ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:15.112010 14526 ts_tablet_manager.cc:1918] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:10:15.112210 14526 log.cc:1196] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/e22976aa256e4cdcaa3274a597475c9f
I20260620 07:10:15.112748 14526 ts_tablet_manager.cc:1939] T e22976aa256e4cdcaa3274a597475c9f P 1dd9e6b7b49f43abaa81b235d451959b: Deleting consensus metadata
I20260620 07:10:15.113584 13934 catalog_manager.cc:4841] TS 1dd9e6b7b49f43abaa81b235d451959b (127.6.53.66:39999): tablet e22976aa256e4cdcaa3274a597475c9f (table test-table [id=abdf94402e4541c4846256eafce0c9d8]) successfully deleted
I20260620 07:10:15.115310 14457 tablet_bootstrap.cc:492] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee: Bootstrap starting.
I20260620 07:10:15.116503 14457 tablet_bootstrap.cc:654] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:15.118218 14457 tablet_bootstrap.cc:492] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee: No bootstrap required, opened a new log
I20260620 07:10:15.118400 14457 ts_tablet_manager.cc:1397] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260620 07:10:15.118942 14457 raft_consensus.cc:348] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:15.119156 14457 raft_consensus.cc:374] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:15.119247 14457 raft_consensus.cc:729] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f0514fc0604b3db4b4385605d73cee, State: Initialized, Role: FOLLOWER
I20260620 07:10:15.119468 14457 consensus_queue.cc:260] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:15.119961 14457 ts_tablet_manager.cc:1428] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 07:10:15.123749 14547 raft_consensus.cc:482] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:10:15.123915 14547 raft_consensus.cc:504] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:15.124563 14547 leader_election.cc:288] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 207fa9f0521d483aa525afb882a2b6b6 (127.6.53.65:43155), 1dd9e6b7b49f43abaa81b235d451959b (127.6.53.66:39999)
I20260620 07:10:15.124984 14107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d367df2bdf24520a1d8ea2478430ece" candidate_uuid: "09f0514fc0604b3db4b4385605d73cee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "207fa9f0521d483aa525afb882a2b6b6" is_pre_election: true
I20260620 07:10:15.125092 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d367df2bdf24520a1d8ea2478430ece" candidate_uuid: "09f0514fc0604b3db4b4385605d73cee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd9e6b7b49f43abaa81b235d451959b" is_pre_election: true
I20260620 07:10:15.125211 14107 raft_consensus.cc:2454] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f0514fc0604b3db4b4385605d73cee in term 0.
I20260620 07:10:15.125361 14243 raft_consensus.cc:2454] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f0514fc0604b3db4b4385605d73cee in term 0.
I20260620 07:10:15.125582 14316 leader_election.cc:302] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f0514fc0604b3db4b4385605d73cee, 207fa9f0521d483aa525afb882a2b6b6; no voters:
I20260620 07:10:15.125799 14547 raft_consensus.cc:2781] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:10:15.125926 14547 raft_consensus.cc:482] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:10:15.126016 14547 raft_consensus.cc:3037] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:15.128151 14547 raft_consensus.cc:504] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:15.128619 14547 leader_election.cc:288] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [CANDIDATE]: Term 1 election: Requested vote from peers 207fa9f0521d483aa525afb882a2b6b6 (127.6.53.65:43155), 1dd9e6b7b49f43abaa81b235d451959b (127.6.53.66:39999)
I20260620 07:10:15.128990 14107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d367df2bdf24520a1d8ea2478430ece" candidate_uuid: "09f0514fc0604b3db4b4385605d73cee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "207fa9f0521d483aa525afb882a2b6b6"
I20260620 07:10:15.129171 14107 raft_consensus.cc:3037] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:15.129132 14243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d367df2bdf24520a1d8ea2478430ece" candidate_uuid: "09f0514fc0604b3db4b4385605d73cee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd9e6b7b49f43abaa81b235d451959b"
I20260620 07:10:15.129323 14243 raft_consensus.cc:3037] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:15.131397 14107 raft_consensus.cc:2454] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f0514fc0604b3db4b4385605d73cee in term 1.
I20260620 07:10:15.131515 14243 raft_consensus.cc:2454] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f0514fc0604b3db4b4385605d73cee in term 1.
I20260620 07:10:15.131899 14316 leader_election.cc:302] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09f0514fc0604b3db4b4385605d73cee, 1dd9e6b7b49f43abaa81b235d451959b; no voters:
I20260620 07:10:15.132372 14547 raft_consensus.cc:2781] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:10:15.132573 14547 raft_consensus.cc:686] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [term 1 LEADER]: Becoming Leader. State: Replica: 09f0514fc0604b3db4b4385605d73cee, State: Running, Role: LEADER
I20260620 07:10:15.132901 14547 consensus_queue.cc:237] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } }
I20260620 07:10:15.135068 13947 catalog_manager.cc:5495] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee reported cstate change: term changed from 0 to 1, leader changed from <none> to 09f0514fc0604b3db4b4385605d73cee (127.6.53.67). New cstate: current_term: 1 leader_uuid: "09f0514fc0604b3db4b4385605d73cee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09f0514fc0604b3db4b4385605d73cee" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 37775 } health_report { overall_health: HEALTHY } } }
I20260620 07:10:15.155927 14243 raft_consensus.cc:1261] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Refusing update from remote peer 09f0514fc0604b3db4b4385605d73cee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:10:15.156019 14107 raft_consensus.cc:1261] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Refusing update from remote peer 09f0514fc0604b3db4b4385605d73cee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:10:15.156452 14474 consensus_queue.cc:1035] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dd9e6b7b49f43abaa81b235d451959b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 39999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:15.156761 14547 consensus_queue.cc:1035] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [LEADER]: Connected to new peer: Peer: permanent_uuid: "207fa9f0521d483aa525afb882a2b6b6" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 43155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:15.195501 13947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:10:15.195742 13947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:10:15.198060 13947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23ad33659ad74a069fecb747845133f3: Sending DeleteTablet for 3 replicas of tablet 7d367df2bdf24520a1d8ea2478430ece
I20260620 07:10:15.198873 14083 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d367df2bdf24520a1d8ea2478430ece with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:15 UTC) from {username='oryx', principal='oryx/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:56568
I20260620 07:10:15.198917 14223 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d367df2bdf24520a1d8ea2478430ece with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:15 UTC) from {username='oryx', principal='oryx/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:42452
I20260620 07:10:15.199065 14361 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d367df2bdf24520a1d8ea2478430ece with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:15 UTC) from {username='oryx', principal='oryx/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:34912
I20260620 07:10:15.199160 14525 tablet_replica.cc:330] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6: stopping tablet replica
I20260620 07:10:15.199199 14526 tablet_replica.cc:330] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b: stopping tablet replica
I20260620 07:10:15.199306 14527 tablet_replica.cc:330] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee: stopping tablet replica
I20260620 07:10:15.199347 14525 raft_consensus.cc:2229] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:10:15.199374 14526 raft_consensus.cc:2229] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:10:15.199467 14527 raft_consensus.cc:2229] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:10:15.199553 14526 raft_consensus.cc:2258] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:15.199550 14525 raft_consensus.cc:2258] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:15.199759 14527 raft_consensus.cc:2258] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:15.200814 14526 ts_tablet_manager.cc:1905] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:15.201182 14527 ts_tablet_manager.cc:1905] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:15.201190 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 14018
I20260620 07:10:15.203121 14525 ts_tablet_manager.cc:1905] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:15.207279 14526 ts_tablet_manager.cc:1918] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:10:15.207418 14526 log.cc:1196] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/7d367df2bdf24520a1d8ea2478430ece
I20260620 07:10:15.207831 14526 ts_tablet_manager.cc:1939] T 7d367df2bdf24520a1d8ea2478430ece P 1dd9e6b7b49f43abaa81b235d451959b: Deleting consensus metadata
I20260620 07:10:15.209271 14527 ts_tablet_manager.cc:1918] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:10:15.209443 14527 log.cc:1196] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/7d367df2bdf24520a1d8ea2478430ece
I20260620 07:10:15.209468 14525 ts_tablet_manager.cc:1918] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:10:15.209622 14525 log.cc:1196] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/7d367df2bdf24520a1d8ea2478430ece
I20260620 07:10:15.209962 14527 ts_tablet_manager.cc:1939] T 7d367df2bdf24520a1d8ea2478430ece P 09f0514fc0604b3db4b4385605d73cee: Deleting consensus metadata
I20260620 07:10:15.209987 14525 ts_tablet_manager.cc:1939] T 7d367df2bdf24520a1d8ea2478430ece P 207fa9f0521d483aa525afb882a2b6b6: Deleting consensus metadata
I20260620 07:10:15.210623 13934 catalog_manager.cc:4841] TS 1dd9e6b7b49f43abaa81b235d451959b (127.6.53.66:39999): tablet 7d367df2bdf24520a1d8ea2478430ece (table test-table [id=25dbde7ce3434035b4e4c40a58536633]) successfully deleted
I20260620 07:10:15.211203 13932 catalog_manager.cc:4841] TS 09f0514fc0604b3db4b4385605d73cee (127.6.53.67:37775): tablet 7d367df2bdf24520a1d8ea2478430ece (table test-table [id=25dbde7ce3434035b4e4c40a58536633]) successfully deleted
I20260620 07:10:15.211494 13934 catalog_manager.cc:4841] TS 207fa9f0521d483aa525afb882a2b6b6 (127.6.53.65:43155): tablet 7d367df2bdf24520a1d8ea2478430ece (table test-table [id=25dbde7ce3434035b4e4c40a58536633]) successfully deleted
Jun 20 07:10:15 dist-test-slave-4n6h krb5kdc[13902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.53.65@KRBTEST.COM
I20260620 07:10:15.225414 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:15.202362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.6.53.65:43155)
0620 07:10:15.209004 (+ 6642us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:15.209010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:15.211614 (+ 2604us) server_negotiation.cc:407] Connection header received
0620 07:10:15.211660 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:15.211664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:15.211720 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:15.211798 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:15.212951 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:15.213632 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:15.214785 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:15.215175 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:15.215265 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:15.219927 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:15.219947 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:15.219963 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:15.219991 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:15.222822 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:15.223472 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:15.223478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:15.223481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:15.223608 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:15.223988 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:15.223992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:15.223994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:15.224288 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:15.224587 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:15.224881 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 07:10:15.225094 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6507,"thread_start_us":6396,"threads_started":1}
I20260620 07:10:15.225993 14149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:15.321439 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 14018
I20260620 07:10:15.339205 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 14158
Jun 20 07:10:15 dist-test-slave-4n6h krb5kdc[13902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.53.66@KRBTEST.COM
I20260620 07:10:15.352697 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:15.339905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.6.53.66:39999)
0620 07:10:15.340376 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:15.340381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:15.340396 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:10:15.340447 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:15.340450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:15.340505 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:15.340592 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:15.341713 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:15.342429 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:15.343550 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:15.343910 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:15.343993 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:15.347751 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:15.347762 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:15.347776 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:15.347806 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:15.350240 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:15.350873 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:15.350879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:15.350881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:15.351015 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:15.351312 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:15.351316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:15.351318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:15.351636 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:15.351864 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:15.352141 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 07:10:15.352312 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":239,"threads_started":1}
I20260620 07:10:15.353387 14288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:15.475247 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 14158
I20260620 07:10:15.498891 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 14297
Jun 20 07:10:15 dist-test-slave-4n6h krb5kdc[13902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.53.67@KRBTEST.COM
I20260620 07:10:15.514513 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:15.499658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.6.53.67:37775)
0620 07:10:15.499826 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:15.499831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:15.499847 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:10:15.500024 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:15.500027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:15.500082 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:15.500162 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:15.501716 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:15.502778 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:15.504687 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:15.505115 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:15.505207 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:15.509478 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:15.509491 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:15.509506 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:15.509543 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:15.512248 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:15.512822 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:15.512828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:15.512830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:15.512962 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:15.513254 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:15.513257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:15.513258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:15.513597 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:15.513812 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:15.514010 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 07:10:15.514155 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:10:15.515141 14427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:15.634912 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 14297
I20260620 07:10:15.656581 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 13915
I20260620 07:10:15.657709 13987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:15.769927 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 13915
2026-06-20T07:10:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5842 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:10:15 dist-test-slave-4n6h krb5kdc[14565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:10:15 dist-test-slave-4n6h krb5kdc[14565](info): set up 2 sockets
Jun 20 07:10:15 dist-test-slave-4n6h krb5kdc[14565](info): commencing operation
krb5kdc: starting...
W20260620 07:10:16.859467 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:10:16 dist-test-slave-4n6h krb5kdc[14565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:10:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:10:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:10:17.067525 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:37193
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:46543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:37193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:17.377591 14578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:17.377933 14578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:17.378106 14578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:17.385265 14578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:10:17.385411 14578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:17.385525 14578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:17.385627 14578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:10:17.385701 14578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:10:17.395344 14578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:46543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:37193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:37193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:17.396571 14578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:17.397795 14578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:17.409026 14586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:17.409103 14583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:17.409067 14584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:17.410001 14578 server_base.cc:1034] running on GCE node
I20260620 07:10:17.410838 14578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:17.412106 14578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:17.413317 14578 hybrid_clock.cc:648] HybridClock initialized: now 1781939417413277 us; error 26 us; skew 500 ppm
I20260620 07:10:17.413739 14578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T07:10:17Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1711 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:10:17 dist-test-slave-4n6h krb5kdc[14592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:10:17 dist-test-slave-4n6h krb5kdc[14592](info): set up 2 sockets
Jun 20 07:10:17 dist-test-slave-4n6h krb5kdc[14592](info): commencing operation
krb5kdc: starting...
W20260620 07:10:18.599190 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:10:18 dist-test-slave-4n6h krb5kdc[14592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:10:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:10:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:10:18.802022 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:42251
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:42251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:19.108697 14605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:19.108952 14605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:19.109099 14605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:19.116547 14605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:10:19.116657 14605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:19.116703 14605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:19.116777 14605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:10:19.116863 14605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:10:19.126322 14605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:42251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:42251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:19.127575 14605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:19.128775 14605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:19.139986 14611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:19.140090 14610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:19.140084 14613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:19.143760 14605 server_base.cc:1034] running on GCE node
I20260620 07:10:19.144407 14605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:19.145617 14605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:19.146819 14605 hybrid_clock.cc:648] HybridClock initialized: now 1781939419146769 us; error 41 us; skew 500 ppm
I20260620 07:10:19.147158 14605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:19 dist-test-slave-4n6h krb5kdc[14592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939419, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:19.150009 14605 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:10:19.151544 14605 webserver.cc:460] Webserver started at http://127.6.53.126:33017/ using document root <none> and password file <none>
I20260620 07:10:19.152195 14605 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:19.152335 14605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:19.152624 14605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:19.155435 14605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "7d7e69223a0a4cf899a63d7e362548a0"
format_stamp: "Formatted at 2026-06-20 07:10:19 on dist-test-slave-4n6h"
server_key: "b31f523c4ae6b3b8a9a7100667ab468d"
server_key_iv: "c346ce37523a6cdbf94dd7f749531891"
server_key_version: "encryptionkey@0"
I20260620 07:10:19.156185 14605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "7d7e69223a0a4cf899a63d7e362548a0"
format_stamp: "Formatted at 2026-06-20 07:10:19 on dist-test-slave-4n6h"
server_key: "b31f523c4ae6b3b8a9a7100667ab468d"
server_key_iv: "c346ce37523a6cdbf94dd7f749531891"
server_key_version: "encryptionkey@0"
I20260620 07:10:19.160804 14605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:10:19.163951 14620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:19.165066 14605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:10:19.165261 14605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "7d7e69223a0a4cf899a63d7e362548a0"
format_stamp: "Formatted at 2026-06-20 07:10:19 on dist-test-slave-4n6h"
server_key: "b31f523c4ae6b3b8a9a7100667ab468d"
server_key_iv: "c346ce37523a6cdbf94dd7f749531891"
server_key_version: "encryptionkey@0"
I20260620 07:10:19.165442 14605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:19.204690 14605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:19.208722 14605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:19.208997 14605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:19.226719 14605 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:42251
I20260620 07:10:19.226747 14671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:42251 every 8 connection(s)
I20260620 07:10:19.227945 14605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:10:19.229768 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 14605
I20260620 07:10:19.229966 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:10:19.230794 6357 external_mini_cluster.cc:1428] Setting key 9935781660cc9992838d3a2c4d816ca7
I20260620 07:10:19.232124 14672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:19.244773 14672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0: Bootstrap starting.
Jun 20 07:10:19 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:10:19.247694 14672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:19.248719 14672 log.cc:824] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:19.251474 14672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0: No bootstrap required, opened a new log
I20260620 07:10:19.255228 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:19.232695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.6.53.126:42251)
0620 07:10:19.236765 (+ 4070us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:19.236776 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:19.236810 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:10:19.239084 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:19.239107 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:19.239460 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:19.239835 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:19.241194 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:19.242488 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:19.243692 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:19.244325 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:19.244520 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:19.248291 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:19.248299 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:19.248312 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:19.248345 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:19.251386 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:19.252053 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:19.252065 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:19.252070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:19.252222 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:19.252621 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:19.252628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:19.252631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:19.253120 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:19.253430 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:19.253870 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 07:10:19.254301 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3842,"thread_start_us":3641,"threads_started":1}
I20260620 07:10:19.259251 14672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7e69223a0a4cf899a63d7e362548a0" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42251 } }
I20260620 07:10:19.259639 14672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:19.259768 14672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7e69223a0a4cf899a63d7e362548a0, State: Initialized, Role: FOLLOWER
I20260620 07:10:19.260303 14672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7e69223a0a4cf899a63d7e362548a0" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42251 } }
I20260620 07:10:19.260535 14672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:10:19.260679 14672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:10:19.260833 14672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:19.262903 14672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7e69223a0a4cf899a63d7e362548a0" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42251 } }
I20260620 07:10:19.263379 14672 leader_election.cc:302] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d7e69223a0a4cf899a63d7e362548a0; no voters:
I20260620 07:10:19.263931 14672 leader_election.cc:288] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:10:19.264070 14677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:10:19.264747 14677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 1 LEADER]: Becoming Leader. State: Replica: 7d7e69223a0a4cf899a63d7e362548a0, State: Running, Role: LEADER
I20260620 07:10:19.265233 14677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7e69223a0a4cf899a63d7e362548a0" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42251 } }
I20260620 07:10:19.266026 14672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:10:19.268007 14678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d7e69223a0a4cf899a63d7e362548a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7e69223a0a4cf899a63d7e362548a0" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42251 } } }
I20260620 07:10:19.268343 14678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:19.270167 14679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d7e69223a0a4cf899a63d7e362548a0. Latest consensus state: current_term: 1 leader_uuid: "7d7e69223a0a4cf899a63d7e362548a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7e69223a0a4cf899a63d7e362548a0" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42251 } } }
I20260620 07:10:19.270409 14682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:10:19.270529 14679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:19.275537 14682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:10:19.283831 14682 catalog_manager.cc:1347] Generated new cluster ID: 88c590b928b24742afc4b6f9ac77c6b1
I20260620 07:10:19.283931 14682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:10:19.297659 14682 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:10:19.299355 14682 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:10:19.313040 14682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0: Generated new TSK 0
I20260620 07:10:19.313963 14682 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:10:19.367621 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42251
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:19.687294 14698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:19.687680 14698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:19.687870 14698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:19.695313 14698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:19.695442 14698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:19.695647 14698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:10:19.705212 14698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:19.706511 14698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:19.707757 14698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:19.720790 14703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:19.720763 14704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:19.721086 14706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:19.721740 14698 server_base.cc:1034] running on GCE node
I20260620 07:10:19.722357 14698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:19.723259 14698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:19.724486 14698 hybrid_clock.cc:648] HybridClock initialized: now 1781939419724431 us; error 46 us; skew 500 ppm
I20260620 07:10:19.724785 14698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:19 dist-test-slave-4n6h krb5kdc[14592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939419, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:19.732475 14698 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:10:19.733853 14698 webserver.cc:460] Webserver started at http://127.6.53.65:37525/ using document root <none> and password file <none>
I20260620 07:10:19.734565 14698 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:19.734706 14698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:19.735018 14698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:19.737844 14698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "98a47406107746ecbc8d35a4229b60fb"
format_stamp: "Formatted at 2026-06-20 07:10:19 on dist-test-slave-4n6h"
server_key: "99013892984a05a03be1fb030cfcf463"
server_key_iv: "01f74d31c9f1135182d519784493c19e"
server_key_version: "encryptionkey@0"
I20260620 07:10:19.738619 14698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "98a47406107746ecbc8d35a4229b60fb"
format_stamp: "Formatted at 2026-06-20 07:10:19 on dist-test-slave-4n6h"
server_key: "99013892984a05a03be1fb030cfcf463"
server_key_iv: "01f74d31c9f1135182d519784493c19e"
server_key_version: "encryptionkey@0"
I20260620 07:10:19.743299 14698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:10:19.746461 14713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:19.747649 14698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:10:19.747834 14698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "98a47406107746ecbc8d35a4229b60fb"
format_stamp: "Formatted at 2026-06-20 07:10:19 on dist-test-slave-4n6h"
server_key: "99013892984a05a03be1fb030cfcf463"
server_key_iv: "01f74d31c9f1135182d519784493c19e"
server_key_version: "encryptionkey@0"
I20260620 07:10:19.748019 14698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:19.763191 14698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:19.767112 14698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:19.767413 14698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:19.768355 14698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:19.769730 14698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:19.769858 14698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:19.769984 14698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:19.770051 14698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:19.801879 14698 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:41005
I20260620 07:10:19.801931 14825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:41005 every 8 connection(s)
I20260620 07:10:19.803098 14698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:10:19.807322 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 14698
I20260620 07:10:19.807508 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:10:19.808118 6357 external_mini_cluster.cc:1428] Setting key b32b12b8b2602f8a11cbd12926d6de49
Jun 20 07:10:19 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939419, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:10:19.823012 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:19.805449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:47363 (local address 127.6.53.126:42251)
0620 07:10:19.805830 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:19.805835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:19.806776 (+ 941us) server_negotiation.cc:407] Connection header received
0620 07:10:19.808620 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:19.808626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:19.808715 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:19.808809 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:19.810625 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:19.811839 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:19.813433 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:19.814128 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:19.814297 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:19.817726 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:19.817731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:19.817736 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:19.817765 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:19.820166 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:19.820848 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:19.820857 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:19.820859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:19.820971 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:19.821391 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:19.821395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:19.821399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:19.821646 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:19.821796 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:19.822447 (+ 651us) server_negotiation.cc:299] Negotiation successful
0620 07:10:19.822652 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":185,"threads_started":1}
I20260620 07:10:19.823882 14829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:19.805937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:47363)
0620 07:10:19.806629 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:19.806663 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:19.808359 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:19.809043 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:19.809051 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:19.809721 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:19.810467 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:19.810485 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:19.811985 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:19.811988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:19.813301 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:19.813313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:19.814370 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:19.814375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:19.814565 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:19.815222 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:19.815243 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:19.817519 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:19.820311 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:19.820320 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:19.820333 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:19.820715 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:19.821076 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:19.821082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:19.821086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:19.821275 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:19.821776 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:19.821783 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:19.822077 (+ 294us) client_negotiation.cc:769] Sending connection context
0620 07:10:19.822365 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 07:10:19.822852 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":294,"threads_started":1}
I20260620 07:10:19.825361 14826 heartbeater.cc:347] Connected to a master server at 127.6.53.126:42251
I20260620 07:10:19.825639 14826 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:19.826383 14826 heartbeater.cc:511] Master 127.6.53.126:42251 requested a full tablet report, sending...
I20260620 07:10:19.828564 14637 ts_manager.cc:194] Registered new tserver with Master: 98a47406107746ecbc8d35a4229b60fb (127.6.53.65:41005)
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
I20260620 07:10:19.830096 14637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:47363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:10:19.859428 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42251
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:20.172084 14832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:20.172389 14832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:20.172554 14832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:20.179910 14832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:20.180048 14832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:20.180217 14832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:10:20.189476 14832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:20.190655 14832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:20.191869 14832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:20.204173 14838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:20.204137 14840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:20.204140 14837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:20.205221 14832 server_base.cc:1034] running on GCE node
I20260620 07:10:20.205787 14832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:20.206827 14832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:20.208043 14832 hybrid_clock.cc:648] HybridClock initialized: now 1781939420208014 us; error 61 us; skew 500 ppm
I20260620 07:10:20.208374 14832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:20 dist-test-slave-4n6h krb5kdc[14592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939420, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:20.211441 14832 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:10:20.212816 14832 webserver.cc:460] Webserver started at http://127.6.53.66:37661/ using document root <none> and password file <none>
I20260620 07:10:20.213441 14832 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:20.213536 14832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:20.213781 14832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:20.216459 14832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "fa6fcd6fe12048e29a7f61dec4312a1f"
format_stamp: "Formatted at 2026-06-20 07:10:20 on dist-test-slave-4n6h"
server_key: "34470093727822b5f27de60c66647a0b"
server_key_iv: "52d2697e42b93c243631b4624a19d762"
server_key_version: "encryptionkey@0"
I20260620 07:10:20.217146 14832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "fa6fcd6fe12048e29a7f61dec4312a1f"
format_stamp: "Formatted at 2026-06-20 07:10:20 on dist-test-slave-4n6h"
server_key: "34470093727822b5f27de60c66647a0b"
server_key_iv: "52d2697e42b93c243631b4624a19d762"
server_key_version: "encryptionkey@0"
I20260620 07:10:20.221889 14832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 07:10:20.225104 14847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:20.226438 14832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:10:20.226606 14832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "fa6fcd6fe12048e29a7f61dec4312a1f"
format_stamp: "Formatted at 2026-06-20 07:10:20 on dist-test-slave-4n6h"
server_key: "34470093727822b5f27de60c66647a0b"
server_key_iv: "52d2697e42b93c243631b4624a19d762"
server_key_version: "encryptionkey@0"
I20260620 07:10:20.226850 14832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:20.255110 14832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:20.259409 14832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:20.259693 14832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:20.260593 14832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:20.261833 14832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:20.261934 14832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:20.262081 14832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:20.262153 14832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:20.295275 14832 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:34915
I20260620 07:10:20.295324 14959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:34915 every 8 connection(s)
I20260620 07:10:20.296473 14832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:10:20.298789 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 14832
I20260620 07:10:20.298957 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:10:20.299486 6357 external_mini_cluster.cc:1428] Setting key 1e6d2ab95852089fd857cc264c4e5021
Jun 20 07:10:20 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939420, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:10:20.317257 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:20.298984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:43419 (local address 127.6.53.126:42251)
0620 07:10:20.299170 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:20.299175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:20.300443 (+ 1268us) server_negotiation.cc:407] Connection header received
0620 07:10:20.302706 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:20.302712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:20.302806 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:20.302919 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:20.304290 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:20.305004 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:20.306589 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:20.306994 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:20.307130 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:20.310677 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:20.310685 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:20.310693 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:20.310737 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:20.314295 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:20.314958 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:20.314966 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:20.314973 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:20.315132 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:20.315575 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:20.315583 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:20.315590 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:20.315962 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:20.316112 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:20.316705 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 07:10:20.316932 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 07:10:20.318145 14962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:20.299472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:43419)
0620 07:10:20.300262 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:20.300327 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:20.302458 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:20.303136 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:20.303144 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:20.303612 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:20.304127 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:20.304142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:20.305149 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:20.305160 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:20.306445 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:20.306462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:20.307208 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:20.307217 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:20.307443 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:20.308110 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:20.308130 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:20.310469 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:20.314419 (+ 3950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:20.314425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:20.314435 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:20.314824 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:20.315235 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:20.315238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:20.315240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:20.315439 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:20.316086 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:20.316092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:20.316358 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 07:10:20.316608 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 07:10:20.317093 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":308,"threads_started":1}
I20260620 07:10:20.319551 14960 heartbeater.cc:347] Connected to a master server at 127.6.53.126:42251
I20260620 07:10:20.319850 14960 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
I20260620 07:10:20.320509 14960 heartbeater.cc:511] Master 127.6.53.126:42251 requested a full tablet report, sending...
I20260620 07:10:20.321992 14637 ts_manager.cc:194] Registered new tserver with Master: fa6fcd6fe12048e29a7f61dec4312a1f (127.6.53.66:34915)
I20260620 07:10:20.322929 14637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:43419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:10:20.347694 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42251
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:20.658773 14965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:20.659186 14965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:20.659344 14965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:20.666707 14965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:20.666857 14965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:20.667057 14965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:10:20.677022 14965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:20.678332 14965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:20.679591 14965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:20.692334 14971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:20.692410 14970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:20.692389 14973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:20.693089 14965 server_base.cc:1034] running on GCE node
I20260620 07:10:20.693638 14965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:20.694551 14965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:20.695782 14965 hybrid_clock.cc:648] HybridClock initialized: now 1781939420695746 us; error 55 us; skew 500 ppm
I20260620 07:10:20.696079 14965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:20 dist-test-slave-4n6h krb5kdc[14592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939420, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:20.698992 14965 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:10:20.700371 14965 webserver.cc:460] Webserver started at http://127.6.53.67:41949/ using document root <none> and password file <none>
I20260620 07:10:20.700996 14965 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:20.701136 14965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:20.701447 14965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:20.704288 14965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "29001d23ab46465ca96fd496fe071d0c"
format_stamp: "Formatted at 2026-06-20 07:10:20 on dist-test-slave-4n6h"
server_key: "6aa0a95e709a61a3f9fe4b20ef2288a1"
server_key_iv: "4bc43de173b6e49195fa94be8f8d2528"
server_key_version: "encryptionkey@0"
I20260620 07:10:20.705044 14965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "29001d23ab46465ca96fd496fe071d0c"
format_stamp: "Formatted at 2026-06-20 07:10:20 on dist-test-slave-4n6h"
server_key: "6aa0a95e709a61a3f9fe4b20ef2288a1"
server_key_iv: "4bc43de173b6e49195fa94be8f8d2528"
server_key_version: "encryptionkey@0"
I20260620 07:10:20.709743 14965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:10:20.713012 14980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:20.714362 14965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:10:20.714515 14965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "29001d23ab46465ca96fd496fe071d0c"
format_stamp: "Formatted at 2026-06-20 07:10:20 on dist-test-slave-4n6h"
server_key: "6aa0a95e709a61a3f9fe4b20ef2288a1"
server_key_iv: "4bc43de173b6e49195fa94be8f8d2528"
server_key_version: "encryptionkey@0"
I20260620 07:10:20.714709 14965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:20.757381 14965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:20.761276 14965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:20.761580 14965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:20.762552 14965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:20.763932 14965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:20.764038 14965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:20.764194 14965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:20.764273 14965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:20.797479 14965 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:39353
I20260620 07:10:20.797513 15092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:39353 every 8 connection(s)
I20260620 07:10:20.798944 14965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:10:20.806667 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 14965
I20260620 07:10:20.806862 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:10:20.807320 6357 external_mini_cluster.cc:1428] Setting key 408a83745ab04b89d3d4610ac508a28b
Jun 20 07:10:20 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939420, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:10:20.818979 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:20.801685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:45189 (local address 127.6.53.126:42251)
0620 07:10:20.801913 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:20.801919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:20.803078 (+ 1159us) server_negotiation.cc:407] Connection header received
0620 07:10:20.805354 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:20.805360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:20.805455 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:20.805564 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:20.807124 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:20.808025 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:20.809532 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:20.809967 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:20.810102 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:20.813672 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:20.813677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:20.813680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:20.813710 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:20.816144 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:20.816869 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:20.816872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:20.816875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:20.816990 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:20.817344 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:20.817347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:20.817349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:20.817657 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:20.817808 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:20.818480 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 07:10:20.818679 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 07:10:20.820029 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:20.802100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:45189)
0620 07:10:20.802924 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:20.802974 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:20.805087 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:20.805790 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:20.805797 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:20.806353 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:20.806951 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:20.806974 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:20.808186 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:20.808191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:20.809383 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:20.809394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:20.810183 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:20.810192 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:20.810466 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:20.811077 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:20.811098 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:20.813358 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:20.816276 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:20.816283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:20.816301 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:20.816759 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:20.817079 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:20.817082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:20.817084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:20.817246 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:20.817773 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:20.817778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:20.818127 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 07:10:20.818395 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 07:10:20.818819 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":381,"threads_started":1}
I20260620 07:10:20.821395 15093 heartbeater.cc:347] Connected to a master server at 127.6.53.126:42251
I20260620 07:10:20.821695 15093 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:20.822430 15093 heartbeater.cc:511] Master 127.6.53.126:42251 requested a full tablet report, sending...
I20260620 07:10:20.823745 14637 ts_manager.cc:194] Registered new tserver with Master: 29001d23ab46465ca96fd496fe071d0c (127.6.53.67:39353)
I20260620 07:10:20.824466 14637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:45189
I20260620 07:10:20.833225 14826 heartbeater.cc:503] Master 127.6.53.126:42251 was elected leader, sending a full tablet report...
I20260620 07:10:20.834511 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.65.keytab.
Entry for principal oryx/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.65.keytab.
WARNING: no policy specified for oryx/127.6.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.66.keytab.
Entry for principal oryx/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.66.keytab.
WARNING: no policy specified for oryx/127.6.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.67.keytab.
Entry for principal oryx/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.67.keytab.
I20260620 07:10:20.965382 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 14698
Jun 20 07:10:20 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:10:20.981156 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:20.966627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.6.53.65:41005)
0620 07:10:20.967280 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:20.967286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:20.967313 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:10:20.967371 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:20.967377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:20.967580 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:20.967703 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:20.969054 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:20.970099 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:20.971388 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:20.971929 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:20.972023 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:20.975960 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:20.975971 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:20.975984 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:20.976016 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:20.978645 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:20.979254 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:20.979304 (+ 50us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:20.979306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:20.979427 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:20.979788 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:20.979802 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:20.979804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:20.980184 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:20.980428 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:20.980631 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 07:10:20.980830 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":344,"threads_started":1}
I20260620 07:10:20.982482 14821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:21.063666 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 14698
I20260620 07:10:21.080323 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 14832
Jun 20 07:10:21 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:10:21.095386 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:21.081117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44038 (local address 127.6.53.66:34915)
0620 07:10:21.081620 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:21.081627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:21.081650 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:10:21.081706 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:21.081711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:21.081903 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:21.082070 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:21.083300 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:21.084319 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:21.085622 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:21.086136 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:21.086327 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:21.090492 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:21.090503 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:21.090516 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:21.090545 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:21.093096 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:21.093669 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:21.093675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:21.093677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:21.093784 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:21.094089 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:21.094092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:21.094093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:21.094468 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:21.094690 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:21.094900 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 07:10:21.095080 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":269,"threads_started":1}
I20260620 07:10:21.096586 14955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:21.177888 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 14832
I20260620 07:10:21.193993 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 14965
Jun 20 07:10:21 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:10:21.209347 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:21.195044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.6.53.67:39353)
0620 07:10:21.195643 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:21.195650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:21.195677 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:10:21.195733 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:21.195739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:21.195968 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:21.196092 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:21.197266 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:21.198234 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:21.199493 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:21.200036 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:21.200130 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:21.204310 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:21.204320 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:21.204334 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:21.204366 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:21.207131 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:21.207725 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:21.207732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:21.207734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:21.207857 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:21.208155 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:21.208158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:21.208160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:21.208445 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:21.208698 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:21.208875 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 07:10:21.209057 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":316,"threads_started":1}
I20260620 07:10:21.210601 15088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:21.296212 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 14965
I20260620 07:10:21.311842 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 14605
I20260620 07:10:21.312933 14667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:21.416525 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 14605
I20260620 07:10:21.436591 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:42251
--webserver_interface=127.6.53.126
--webserver_port=33017
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:42251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:21.744853 15110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:21.745102 15110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:21.745307 15110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:21.752357 15110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:10:21.752478 15110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:21.752561 15110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:21.752667 15110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:10:21.752748 15110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:10:21.762075 15110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:42251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:42251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=33017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:21.763254 15110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:21.764487 15110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:21.775807 15115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:21.775784 15118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:21.775749 15116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:21.777009 15110 server_base.cc:1034] running on GCE node
I20260620 07:10:21.777825 15110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:21.779147 15110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:21.780375 15110 hybrid_clock.cc:648] HybridClock initialized: now 1781939421780330 us; error 61 us; skew 500 ppm
I20260620 07:10:21.780701 15110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:21 dist-test-slave-4n6h krb5kdc[14592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939421, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:21.783761 15110 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:10:21.785167 15110 webserver.cc:460] Webserver started at http://127.6.53.126:33017/ using document root <none> and password file <none>
I20260620 07:10:21.785825 15110 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:21.785982 15110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:21.791000 15110 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:10:21.793589 15125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:21.794891 15110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:10:21.795059 15110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "7d7e69223a0a4cf899a63d7e362548a0"
format_stamp: "Formatted at 2026-06-20 07:10:19 on dist-test-slave-4n6h"
server_key: "b31f523c4ae6b3b8a9a7100667ab468d"
server_key_iv: "c346ce37523a6cdbf94dd7f749531891"
server_key_version: "encryptionkey@0"
I20260620 07:10:21.795980 15110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:21.812778 15110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:21.817070 15110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:21.817344 15110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:21.835512 15110 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:42251
I20260620 07:10:21.835521 15176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:42251 every 8 connection(s)
I20260620 07:10:21.837656 15110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:10:21.845630 15177 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:10:21.845620 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 15110
I20260620 07:10:21.846441 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:41005
--local_ip_for_outbound_sockets=127.6.53.65
--tserver_master_addrs=127.6.53.126:42251
--webserver_port=37525
--webserver_interface=127.6.53.65
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.53.65
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:10:21.848769 15177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0: Bootstrap starting.
I20260620 07:10:21.867252 15177 log.cc:824] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:21.875614 15177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:10:21.876108 15177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0: Bootstrap complete.
I20260620 07:10:21.884967 15177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7e69223a0a4cf899a63d7e362548a0" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42251 } }
I20260620 07:10:21.885799 15177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7e69223a0a4cf899a63d7e362548a0, State: Initialized, Role: FOLLOWER
I20260620 07:10:21.886461 15177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7e69223a0a4cf899a63d7e362548a0" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42251 } }
I20260620 07:10:21.886705 15177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:10:21.886828 15177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:10:21.887013 15177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:10:21.890594 15177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7e69223a0a4cf899a63d7e362548a0" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42251 } }
I20260620 07:10:21.891053 15177 leader_election.cc:302] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d7e69223a0a4cf899a63d7e362548a0; no voters:
I20260620 07:10:21.891685 15177 leader_election.cc:288] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:10:21.891892 15181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:10:21.893034 15181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [term 2 LEADER]: Becoming Leader. State: Replica: 7d7e69223a0a4cf899a63d7e362548a0, State: Running, Role: LEADER
I20260620 07:10:21.893570 15177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:10:21.893757 15181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7e69223a0a4cf899a63d7e362548a0" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42251 } }
I20260620 07:10:21.896994 15183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d7e69223a0a4cf899a63d7e362548a0. Latest consensus state: current_term: 2 leader_uuid: "7d7e69223a0a4cf899a63d7e362548a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7e69223a0a4cf899a63d7e362548a0" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42251 } } }
I20260620 07:10:21.897365 15183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:21.897970 15182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7d7e69223a0a4cf899a63d7e362548a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7e69223a0a4cf899a63d7e362548a0" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42251 } } }
I20260620 07:10:21.898352 15182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0 [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:21.907171 15191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:10:21.912484 15191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:10:21.913863 15191 catalog_manager.cc:1259] Loaded cluster ID: 88c590b928b24742afc4b6f9ac77c6b1
I20260620 07:10:21.914014 15191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:10:21.917878 15191 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:10:21.919317 15191 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7d7e69223a0a4cf899a63d7e362548a0: Loaded TSK: 0
I20260620 07:10:21.920810 15191 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:10:22.174836 15179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:22.175076 15179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:22.175223 15179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:22.182519 15179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:22.182612 15179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:22.182797 15179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:10:22.192230 15179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.65.keytab
--principal=oryx/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:41005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=37525
--tserver_master_addrs=127.6.53.126:42251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:22.193457 15179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:22.194880 15179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:22.209074 15204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:22.209625 15203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:22.210281 15206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:22.211287 15179 server_base.cc:1034] running on GCE node
I20260620 07:10:22.211840 15179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:22.212765 15179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:22.214025 15179 hybrid_clock.cc:648] HybridClock initialized: now 1781939422213972 us; error 47 us; skew 500 ppm
I20260620 07:10:22.214404 15179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:22 dist-test-slave-4n6h krb5kdc[14592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939422, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:22.217499 15179 init.cc:377] Logged in from keytab as oryx/127.6.53.65@KRBTEST.COM (short username oryx)
I20260620 07:10:22.219121 15179 webserver.cc:460] Webserver started at http://127.6.53.65:37525/ using document root <none> and password file <none>
I20260620 07:10:22.219807 15179 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:22.219964 15179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:22.225034 15179 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 07:10:22.227842 15213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:22.229174 15179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 07:10:22.229331 15179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "98a47406107746ecbc8d35a4229b60fb"
format_stamp: "Formatted at 2026-06-20 07:10:19 on dist-test-slave-4n6h"
server_key: "99013892984a05a03be1fb030cfcf463"
server_key_iv: "01f74d31c9f1135182d519784493c19e"
server_key_version: "encryptionkey@0"
I20260620 07:10:22.230203 15179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:22.249432 15179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:22.253253 15179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:22.253512 15179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:22.254561 15179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:22.255826 15179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:22.255951 15179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:22.256083 15179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:22.256186 15179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:22.288693 15179 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:41005
I20260620 07:10:22.288777 15325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:41005 every 8 connection(s)
I20260620 07:10:22.290107 15179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:10:22.296921 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 15179
I20260620 07:10:22.297681 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:34915
--local_ip_for_outbound_sockets=127.6.53.66
--tserver_master_addrs=127.6.53.126:42251
--webserver_port=37661
--webserver_interface=127.6.53.66
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.53.66
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:10:22 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:22 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:22.308509 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:22.292964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:47705)
0620 07:10:22.293765 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:22.293813 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:22.295499 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:22.296832 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:22.296844 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:22.297508 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:22.298269 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:22.298296 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.299990 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.299998 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:22.302088 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:22.302104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.303135 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.303143 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:22.303448 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:22.304163 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:22.304189 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:22.307685 (+ 3496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":337,"threads_started":1}
W20260620 07:10:22.309154 15326 heartbeater.cc:644] Failed to heartbeat to 127.6.53.126:42251 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.53.126:42251: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
W20260620 07:10:22.309177 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:22.293028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:47705 (local address 127.6.53.126:42251)
0620 07:10:22.293772 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:22.293783 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:22.293932 (+ 149us) server_negotiation.cc:407] Connection header received
0620 07:10:22.295979 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:22.295992 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:22.296330 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:22.296607 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:22.298460 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.299786 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.302226 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.302903 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.303028 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:22.308106 (+ 5078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:47705: BlockingRecv error: recv got EOF from 127.6.53.65:47705 (error 108)
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":312,"threads_started":1}
Jun 20 07:10:22 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:22 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:22.317262 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:22.309751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:53167)
0620 07:10:22.309930 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:22.309945 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:22.310043 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:22.310469 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:22.310474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:22.310795 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:22.310913 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:22.310920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.311927 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.311932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:22.312853 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:22.312864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.313628 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.313633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:22.313801 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:22.314382 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:22.314398 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:22.317080 (+ 2682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 07:10:22.317627 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:22.309965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:53167 (local address 127.6.53.126:42251)
0620 07:10:22.310155 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:22.310161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:22.310176 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:10:22.310223 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:22.310227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:22.310340 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:22.310449 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:22.311047 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.311781 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.312995 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.313484 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.313617 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:22.317359 (+ 3742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:53167: BlockingRecv error: recv got EOF from 127.6.53.65:53167 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 07:10:22 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:22 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:22.325105 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:22.317874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:46419)
0620 07:10:22.318058 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:22.318072 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:22.318141 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:22.318578 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:22.318582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:22.318864 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:22.318988 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:22.318995 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.319923 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.319929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:22.320787 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:22.320797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.321464 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.321469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:22.321651 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:22.322227 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:22.322243 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:22.324829 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 07:10:22.325358 15326 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:10:22.325506 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:22.318156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:46419 (local address 127.6.53.126:42251)
0620 07:10:22.318344 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:22.318348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:22.318360 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:10:22.318393 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:22.318396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:22.318453 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:22.318558 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:22.319122 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.319786 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.320926 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.321319 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.321444 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:22.325018 (+ 3574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:46419: BlockingRecv error: recv got EOF from 127.6.53.65:46419 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 07:10:22.616918 15330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:22.617238 15330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:22.617414 15330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:22.624529 15330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:22.624678 15330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:22.624850 15330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:10:22.634164 15330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.66.keytab
--principal=oryx/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:34915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=37661
--tserver_master_addrs=127.6.53.126:42251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:22.635411 15330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:22.636623 15330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:22.649200 15336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:22.649286 15338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:22.649286 15335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:22.650199 15330 server_base.cc:1034] running on GCE node
I20260620 07:10:22.650831 15330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:22.651736 15330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:22.652984 15330 hybrid_clock.cc:648] HybridClock initialized: now 1781939422652909 us; error 48 us; skew 500 ppm
I20260620 07:10:22.653290 15330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:22 dist-test-slave-4n6h krb5kdc[14592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939422, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:22.656320 15330 init.cc:377] Logged in from keytab as oryx/127.6.53.66@KRBTEST.COM (short username oryx)
I20260620 07:10:22.657858 15330 webserver.cc:460] Webserver started at http://127.6.53.66:37661/ using document root <none> and password file <none>
I20260620 07:10:22.658578 15330 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:22.658756 15330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:22.663640 15330 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:10:22.666028 15345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:22.667333 15330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:10:22.667456 15330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "fa6fcd6fe12048e29a7f61dec4312a1f"
format_stamp: "Formatted at 2026-06-20 07:10:20 on dist-test-slave-4n6h"
server_key: "34470093727822b5f27de60c66647a0b"
server_key_iv: "52d2697e42b93c243631b4624a19d762"
server_key_version: "encryptionkey@0"
I20260620 07:10:22.668318 15330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:22.693145 15330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:22.697716 15330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:22.697924 15330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:22.698899 15330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:22.700249 15330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:22.700335 15330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:22.700409 15330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:22.700464 15330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:22.732996 15330 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:34915
I20260620 07:10:22.733104 15457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:34915 every 8 connection(s)
I20260620 07:10:22.734760 15330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:10:22.740483 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 15330
I20260620 07:10:22.741079 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:39353
--local_ip_for_outbound_sockets=127.6.53.67
--tserver_master_addrs=127.6.53.126:42251
--webserver_port=41949
--webserver_interface=127.6.53.67
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.53.67
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:10:22 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:22 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 07:10:22.750082 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:22.737111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:45723 (local address 127.6.53.126:42251)
0620 07:10:22.737279 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:22.737284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:22.738372 (+ 1088us) server_negotiation.cc:407] Connection header received
0620 07:10:22.740208 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:22.740220 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:22.740320 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:22.740438 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:22.741954 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.742731 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.744346 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.744764 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.744899 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:22.749826 (+ 4927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:45723: BlockingRecv error: recv got EOF from 127.6.53.66:45723 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:10:22.750756 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:22.737549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:45723)
0620 07:10:22.738195 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:22.738228 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:22.739887 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:22.740690 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:22.740700 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:22.741181 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:22.741782 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:22.741799 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.742887 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.742892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:22.744174 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:22.744186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.744986 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.744993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:22.745225 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:22.745895 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:22.745917 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:22.749673 (+ 3756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":257,"threads_started":1}
W20260620 07:10:22.751497 15458 heartbeater.cc:644] Failed to heartbeat to 127.6.53.126:42251 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.53.126:42251: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Jun 20 07:10:22 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:22 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:22.760452 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:22.752163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:60975)
0620 07:10:22.752347 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:22.752362 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:22.752439 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:22.752774 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:22.752777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:22.753063 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:22.753212 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:22.753219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.754734 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.754738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:22.755606 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:22.755616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.756696 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.756700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:22.756811 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:22.757388 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:22.757404 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:22.760261 (+ 2857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 07:10:22.760864 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:22.752268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:60975 (local address 127.6.53.126:42251)
0620 07:10:22.752470 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:22.752478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:22.752498 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:10:22.752551 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:22.752558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:22.752640 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:22.752749 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:22.753353 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.754596 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.755764 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.756536 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.756689 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:22.760580 (+ 3891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:60975: BlockingRecv error: recv got EOF from 127.6.53.66:60975 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 07:10:22 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:22 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:22.768481 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:22.761328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:33221)
0620 07:10:22.761464 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:22.761505 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:22.761598 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:22.761893 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:22.761898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:22.762178 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:22.762378 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:22.762387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.763292 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.763298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:22.764181 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:22.764198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.764856 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.764862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:22.765027 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:22.765597 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:22.765613 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:22.768203 (+ 2590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 07:10:22.768764 15458 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:10:22.768559 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:22.761443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:33221 (local address 127.6.53.126:42251)
0620 07:10:22.761620 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:22.761627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:22.761647 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:10:22.761703 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:22.761708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:22.761772 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:22.761873 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:22.762502 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.763161 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.764328 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:22.764712 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:22.764837 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:22.768357 (+ 3520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:33221: BlockingRecv error: recv got EOF from 127.6.53.66:33221 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 07:10:23.060317 15461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:23.060607 15461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:23.060781 15461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:23.067828 15461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:23.067979 15461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:23.068166 15461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:10:23.077788 15461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/krb5kdc/oryx_127.6.53.67.keytab
--principal=oryx/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:39353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=41949
--tserver_master_addrs=127.6.53.126:42251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:23.079033 15461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:23.080415 15461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:23.093024 15469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:23.093057 15467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:23.093088 15466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:23.093919 15461 server_base.cc:1034] running on GCE node
I20260620 07:10:23.094581 15461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:23.095484 15461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:23.096758 15461 hybrid_clock.cc:648] HybridClock initialized: now 1781939423096711 us; error 61 us; skew 500 ppm
I20260620 07:10:23.097106 15461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:23 dist-test-slave-4n6h krb5kdc[14592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939423, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:23.100121 15461 init.cc:377] Logged in from keytab as oryx/127.6.53.67@KRBTEST.COM (short username oryx)
I20260620 07:10:23.101583 15461 webserver.cc:460] Webserver started at http://127.6.53.67:41949/ using document root <none> and password file <none>
I20260620 07:10:23.102170 15461 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:23.102350 15461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:23.107851 15461 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 07:10:23.110319 15476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:23.111603 15461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:10:23.111742 15461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "29001d23ab46465ca96fd496fe071d0c"
format_stamp: "Formatted at 2026-06-20 07:10:20 on dist-test-slave-4n6h"
server_key: "6aa0a95e709a61a3f9fe4b20ef2288a1"
server_key_iv: "4bc43de173b6e49195fa94be8f8d2528"
server_key_version: "encryptionkey@0"
I20260620 07:10:23.112649 15461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:23.126202 15461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:23.129843 15461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:23.130055 15461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:23.130961 15461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:23.132164 15461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:23.132256 15461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:23.132341 15461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:23.132396 15461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:23.164912 15461 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:39353
I20260620 07:10:23.164999 15588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:39353 every 8 connection(s)
I20260620 07:10:23.166518 15461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:10:23.172612 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 15461
Jun 20 07:10:23 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:23 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 07:10:23.183586 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:23.168851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:60011 (local address 127.6.53.126:42251)
0620 07:10:23.169058 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:23.169064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:23.170093 (+ 1029us) server_negotiation.cc:407] Connection header received
0620 07:10:23.171910 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:23.171914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:23.171979 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:23.172063 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:23.174108 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.175111 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.177247 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.177869 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.177952 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:23.183333 (+ 5381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:60011: BlockingRecv error: recv got EOF from 127.6.53.67:60011 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:10:23.184266 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:23.169290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:60011)
0620 07:10:23.169940 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:23.169982 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:23.171674 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:23.172348 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:23.172360 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:23.173089 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:23.173902 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:23.173928 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.175323 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.175329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:23.177103 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:23.177118 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.178112 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.178121 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:23.178459 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:23.179155 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:23.179190 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:23.183130 (+ 3940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":267,"threads_started":1}
W20260620 07:10:23.184903 15589 heartbeater.cc:644] Failed to heartbeat to 127.6.53.126:42251 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.53.126:42251: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
I20260620 07:10:23.190635 15593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:23.173781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.6.53.126:42251)
0620 07:10:23.174618 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:23.174622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:23.174636 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:10:23.174672 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:23.174675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:23.174720 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:23.174811 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:23.176708 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.177648 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.179583 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.180072 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.180165 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:23.183071 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:23.183084 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:23.183103 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:23.183155 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:23.187174 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:23.187938 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:23.187943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:23.187949 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:23.188206 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:23.188567 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:23.188572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:23.188576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:23.189120 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:23.189416 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:23.189820 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 07:10:23.190193 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":750,"thread_start_us":632,"threads_started":1}
Jun 20 07:10:23 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:23 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:23.194473 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:23.185510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:53177)
0620 07:10:23.185740 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:23.185757 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:23.185866 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:23.186205 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:23.186209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:23.186624 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:23.186804 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:23.186815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.188063 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.188068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:23.189398 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:23.189412 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.190494 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.190500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:23.190689 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:23.191260 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:23.191277 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:23.194153 (+ 2876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112}
W20260620 07:10:23.194607 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:23.185662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:53177 (local address 127.6.53.126:42251)
0620 07:10:23.185856 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:23.185860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:23.185880 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:10:23.186003 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:23.186007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:23.186065 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:23.186146 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:23.186948 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.187925 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.189585 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.190357 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.190455 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:23.194374 (+ 3919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:53177: BlockingRecv error: recv got EOF from 127.6.53.67:53177 (error 108)
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":52}
Jun 20 07:10:23 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:23 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:23.202060 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:23.195042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:41825)
0620 07:10:23.195181 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:23.195196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:23.195265 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:23.195704 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:23.195720 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:23.195951 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:23.196072 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:23.196078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.197040 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.197044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:23.197955 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:23.197965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.198624 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.198628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:23.198767 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:23.199321 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:23.199336 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:23.201791 (+ 2455us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 07:10:23.202204 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:23.195302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:41825 (local address 127.6.53.126:42251)
0620 07:10:23.195494 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:23.195497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:23.195509 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:10:23.195540 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:23.195543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:23.195586 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:23.195652 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:23.196228 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.196936 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.198070 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.198490 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.198575 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:23.201952 (+ 3377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:41825: BlockingRecv error: recv got EOF from 127.6.53.67:41825 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260620 07:10:23.202376 15589 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 07:10:23 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:23 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:23.334069 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:23.325955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:45351)
0620 07:10:23.326481 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:23.326497 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:23.326602 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:23.326880 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:23.326883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:23.327314 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:23.327495 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:23.327503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.328443 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.328446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:23.329358 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:23.329368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.330091 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.330096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:23.330207 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:23.330765 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:23.330782 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:23.333694 (+ 2912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":294,"threads_started":1}
W20260620 07:10:23.334141 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:23.326031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:45351 (local address 127.6.53.126:42251)
0620 07:10:23.326236 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:23.326240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:23.326578 (+ 338us) server_negotiation.cc:407] Connection header received
0620 07:10:23.326690 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:23.326694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:23.326754 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:23.326835 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:23.327626 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.328322 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.329488 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.329941 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.330040 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:23.333919 (+ 3879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:45351: BlockingRecv error: recv got EOF from 127.6.53.65:45351 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 07:10:23 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:23 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:23.781104 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:23.769509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:47217)
0620 07:10:23.769973 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:23.769992 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:23.770108 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:23.770432 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:23.770435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:23.770867 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:23.771054 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:23.771063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.772147 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.772152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:23.773962 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:23.773982 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.775127 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.775136 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:23.775342 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:23.776271 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:23.776297 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:23.780772 (+ 4475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":255,"threads_started":1}
W20260620 07:10:23.781749 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:23.769536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:47217 (local address 127.6.53.126:42251)
0620 07:10:23.769799 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:23.769806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:23.770089 (+ 283us) server_negotiation.cc:407] Connection header received
0620 07:10:23.770205 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:23.770209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:23.770300 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:23.770380 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:23.771195 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.771935 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.774149 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:23.774906 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:23.775059 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:23.781122 (+ 6063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:47217: BlockingRecv error: recv got EOF from 127.6.53.66:47217 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 07:10:24 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:24 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:24.211462 15596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:24.203005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:49945)
0620 07:10:24.203447 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:24.203466 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:24.203576 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:24.203855 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:24.203858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:24.204311 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:24.204497 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:24.204506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:24.205466 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:24.205470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:24.206538 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:24.206549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:24.207486 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:24.207491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:24.207670 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:24.208357 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:24.208373 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:24.211244 (+ 2871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":247,"threads_started":1}
W20260620 07:10:24.211642 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:24.203041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:49945 (local address 127.6.53.126:42251)
0620 07:10:24.203289 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:24.203295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:24.203547 (+ 252us) server_negotiation.cc:407] Connection header received
0620 07:10:24.203676 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:24.203680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:24.203738 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:24.203816 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:24.204613 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:24.205328 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:24.206683 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:24.207291 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:24.207455 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:24.211425 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:49945: BlockingRecv error: recv got EOF from 127.6.53.67:49945 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 07:10:24 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:24 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:24.343174 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:24.335237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:46129)
0620 07:10:24.335750 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:24.335765 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:24.335924 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:24.336275 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:24.336279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:24.336675 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:24.336823 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:24.336832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:24.337862 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:24.337868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:24.338772 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:24.338783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:24.339507 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:24.339512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:24.339682 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:24.340227 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:24.340245 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:24.342932 (+ 2687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":274,"threads_started":1}
W20260620 07:10:24.343370 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:24.335280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:46129 (local address 127.6.53.126:42251)
0620 07:10:24.335481 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:24.335485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:24.335856 (+ 371us) server_negotiation.cc:407] Connection header received
0620 07:10:24.336058 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:24.336064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:24.336134 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:24.336251 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:24.336973 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:24.337717 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:24.338918 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:24.339344 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:24.339492 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:24.343118 (+ 3626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:46129: BlockingRecv error: recv got EOF from 127.6.53.65:46129 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 07:10:24 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:24 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:24.795333 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:24.781967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:50263)
0620 07:10:24.782560 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:24.782576 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:24.782694 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:24.782993 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:24.782997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:24.783476 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:24.783619 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:24.783627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:24.784650 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:24.784654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:24.785537 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:24.785548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:24.786180 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:24.786184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:24.786310 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:24.792110 (+ 5800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:24.792127 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:24.795105 (+ 2978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":357,"threads_started":1}
W20260620 07:10:24.795639 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:24.782011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:50263 (local address 127.6.53.126:42251)
0620 07:10:24.782205 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:24.782209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:24.782661 (+ 452us) server_negotiation.cc:407] Connection header received
0620 07:10:24.782796 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:24.782800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:24.782861 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:24.782955 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:24.783752 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:24.784536 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:24.785675 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:24.786060 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:24.786148 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:24.795297 (+ 9149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:50263: BlockingRecv error: recv got EOF from 127.6.53.66:50263 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 07:10:25 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:25 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:25.220495 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:25.212321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:46313)
0620 07:10:25.212873 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:25.212894 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:25.213006 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:25.213309 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:25.213312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:25.213671 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:25.213810 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:25.213818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:25.214854 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:25.214859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:25.215737 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:25.215749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:25.216453 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:25.216457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:25.216565 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:25.217125 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:25.217141 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:25.220123 (+ 2982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":325,"threads_started":1}
W20260620 07:10:25.220620 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:25.212460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:46313 (local address 127.6.53.126:42251)
0620 07:10:25.212628 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:25.212632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:25.212965 (+ 333us) server_negotiation.cc:407] Connection header received
0620 07:10:25.213101 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:25.213104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:25.213164 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:25.213248 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:25.213931 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:25.214734 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:25.215855 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:25.216328 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:25.216420 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:25.220329 (+ 3909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:46313: BlockingRecv error: recv got EOF from 127.6.53.67:46313 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 07:10:25 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:25 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:25.352967 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:25.344246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:39281)
0620 07:10:25.344765 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:25.344781 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:25.344891 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:25.345240 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:25.345243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:25.345561 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:25.345716 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:25.345724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:25.346760 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:25.346764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:25.348078 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:25.348095 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:25.348860 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:25.348865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:25.349047 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:25.349755 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:25.349773 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:25.352608 (+ 2835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":288,"threads_started":1}
W20260620 07:10:25.353058 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:25.344569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:39281 (local address 127.6.53.126:42251)
0620 07:10:25.344788 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:25.344792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:25.344838 (+ 46us) server_negotiation.cc:407] Connection header received
0620 07:10:25.345020 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:25.345023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:25.345089 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:25.345182 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:25.345872 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:25.346643 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:25.348229 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:25.348713 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:25.348852 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:25.352805 (+ 3953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:39281: BlockingRecv error: recv got EOF from 127.6.53.65:39281 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 07:10:25 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:25 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:25.804167 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:25.796254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:54291)
0620 07:10:25.796774 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:25.796792 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:25.796911 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:25.797243 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:25.797246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:25.797611 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:25.797793 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:25.797802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:25.798831 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:25.798835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:25.799803 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:25.799815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:25.800480 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:25.800484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:25.800612 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:25.801168 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:25.801187 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:25.803927 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":272,"threads_started":1}
W20260620 07:10:25.804370 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:25.796300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:54291 (local address 127.6.53.126:42251)
0620 07:10:25.796453 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:25.796457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:25.796867 (+ 410us) server_negotiation.cc:407] Connection header received
0620 07:10:25.797036 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:25.797040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:25.797100 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:25.797196 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:25.797930 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:25.798703 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:25.799942 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:25.800359 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:25.800474 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:25.804116 (+ 3642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:54291: BlockingRecv error: recv got EOF from 127.6.53.66:54291 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 07:10:26 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:26 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:26.229614 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:26.221537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:34915)
0620 07:10:26.222025 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:26.222047 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:26.222202 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:26.222580 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:26.222586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:26.222913 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:26.223107 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:26.223115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:26.224132 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:26.224136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:26.224986 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:26.224996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:26.225644 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:26.225649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:26.225826 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:26.226479 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:26.226501 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:26.229362 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":257,"threads_started":1}
W20260620 07:10:26.229842 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:26.221591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:34915 (local address 127.6.53.126:42251)
0620 07:10:26.221803 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:26.221812 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:26.222132 (+ 320us) server_negotiation.cc:407] Connection header received
0620 07:10:26.222359 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:26.222363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:26.222426 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:26.222512 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:26.223253 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:26.223967 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:26.225122 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:26.225509 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:26.225633 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:26.229613 (+ 3980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:34915: BlockingRecv error: recv got EOF from 127.6.53.67:34915 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 07:10:26 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:26 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:26.362149 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:26.353896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:39667)
0620 07:10:26.354411 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:26.354427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:26.354584 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:26.354933 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:26.354938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:26.355335 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:26.355477 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:26.355486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:26.356473 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:26.356480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:26.357442 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:26.357453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:26.358186 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:26.358191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:26.358394 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:26.359039 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:26.359057 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:26.361921 (+ 2864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":274,"threads_started":1}
W20260620 07:10:26.362397 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:26.353997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:39667 (local address 127.6.53.126:42251)
0620 07:10:26.354270 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:26.354277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:26.354513 (+ 236us) server_negotiation.cc:407] Connection header received
0620 07:10:26.354720 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:26.354725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:26.354799 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:26.354901 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:26.355621 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:26.356344 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:26.357588 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:26.358014 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:26.358142 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:26.362136 (+ 3994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:39667: BlockingRecv error: recv got EOF from 127.6.53.65:39667 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 07:10:26 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:26 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:26.813390 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:26.805086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:57953)
0620 07:10:26.805573 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:26.805590 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:26.805757 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:26.806102 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:26.806106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:26.806524 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:26.806680 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:26.806690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:26.807696 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:26.807702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:26.808628 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:26.808639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:26.809341 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:26.809347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:26.809536 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:26.810100 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:26.810117 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:26.813099 (+ 2982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":275,"threads_started":1}
W20260620 07:10:26.813637 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:26.805148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:57953 (local address 127.6.53.126:42251)
0620 07:10:26.805382 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:26.805390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:26.805679 (+ 289us) server_negotiation.cc:407] Connection header received
0620 07:10:26.805896 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:26.805901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:26.805960 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:26.806064 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:26.806828 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:26.807545 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:26.808784 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:26.809190 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:26.809318 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:26.813371 (+ 4053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:57953: BlockingRecv error: recv got EOF from 127.6.53.66:57953 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 07:10:27 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:27 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:27.238492 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:27.230542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:43003)
0620 07:10:27.230958 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:27.230977 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:27.231082 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:27.231367 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:27.231371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:27.231730 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:27.231878 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:27.231887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:27.232846 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:27.232850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:27.233894 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:27.233906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:27.234594 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:27.234599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:27.234730 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:27.235268 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:27.235285 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:27.238213 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":234,"threads_started":1}
W20260620 07:10:27.238651 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:27.230605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:43003 (local address 127.6.53.126:42251)
0620 07:10:27.230850 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:27.230855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:27.231054 (+ 199us) server_negotiation.cc:407] Connection header received
0620 07:10:27.231185 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:27.231189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:27.231249 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:27.231328 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:27.231995 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:27.232726 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:27.234044 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:27.234466 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:27.234553 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:27.238441 (+ 3888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:43003: BlockingRecv error: recv got EOF from 127.6.53.67:43003 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 20 07:10:27 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:27 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:27.371477 15606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:27.363181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:40941)
0620 07:10:27.363697 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:27.363714 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:27.363825 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:27.364114 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:27.364117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:27.364533 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:27.364758 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:27.364770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:27.366144 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:27.366148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:27.367051 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:27.367062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:27.367722 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:27.367726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:27.367841 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:27.368375 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:27.368391 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:27.371118 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":249,"threads_started":1}
W20260620 07:10:27.371585 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:27.363280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:40941 (local address 127.6.53.126:42251)
0620 07:10:27.363544 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:27.363550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:27.363792 (+ 242us) server_negotiation.cc:407] Connection header received
0620 07:10:27.363925 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:27.363928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:27.363991 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:27.364070 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:27.364930 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:27.366013 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:27.367180 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:27.367588 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:27.367692 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:27.371318 (+ 3626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:40941: BlockingRecv error: recv got EOF from 127.6.53.65:40941 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 20 07:10:27 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:27 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:27.822077 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:27.814328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:49457)
0620 07:10:27.814855 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:27.814870 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:27.814972 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:27.815266 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:27.815269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:27.815726 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:27.815882 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:27.815891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:27.816921 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:27.816925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:27.817814 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:27.817826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:27.818546 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:27.818550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:27.818684 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:27.819216 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:27.819232 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:27.821842 (+ 2610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":250,"threads_started":1}
W20260620 07:10:27.822299 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:27.814562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:49457 (local address 127.6.53.126:42251)
0620 07:10:27.814736 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:27.814740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:27.814936 (+ 196us) server_negotiation.cc:407] Connection header received
0620 07:10:27.815081 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:27.815085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:27.815142 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:27.815225 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:27.816000 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:27.816791 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:27.817950 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:27.818427 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:27.818519 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:27.822025 (+ 3506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:49457: BlockingRecv error: recv got EOF from 127.6.53.66:49457 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 07:10:28 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:28 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:28.247385 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:28.239386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:37929)
0620 07:10:28.239924 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:28.239961 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:28.240066 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:28.240413 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:28.240416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:28.240831 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:28.240984 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:28.240992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:28.241987 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:28.241991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:28.242904 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:28.242916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:28.243514 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:28.243517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:28.243650 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:28.244212 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:28.244228 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:28.247052 (+ 2824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":292,"threads_started":1}
W20260620 07:10:28.247522 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:28.239481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:37929 (local address 127.6.53.126:42251)
0620 07:10:28.239650 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:28.239656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:28.240035 (+ 379us) server_negotiation.cc:407] Connection header received
0620 07:10:28.240172 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:28.240175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:28.240273 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:28.240364 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:28.241106 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:28.241853 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:28.243016 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:28.243399 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:28.243487 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:28.247277 (+ 3790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:37929: BlockingRecv error: recv got EOF from 127.6.53.67:37929 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 07:10:28 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:28 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:28.380390 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:28.372431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:54905)
0620 07:10:28.373040 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:28.373061 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:28.373189 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:28.373486 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:28.373489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:28.373844 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:28.373981 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:28.373989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:28.375058 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:28.375062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:28.375923 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:28.375934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:28.376563 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:28.376568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:28.376700 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:28.377352 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:28.377376 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:28.380182 (+ 2806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":337,"threads_started":1}
W20260620 07:10:28.380568 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:28.372523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:54905 (local address 127.6.53.126:42251)
0620 07:10:28.372712 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:28.372716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:28.373152 (+ 436us) server_negotiation.cc:407] Connection header received
0620 07:10:28.373295 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:28.373299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:28.373360 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:28.373444 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:28.374100 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:28.374932 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:28.376037 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:28.376434 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:28.376525 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:28.380343 (+ 3818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:54905: BlockingRecv error: recv got EOF from 127.6.53.65:54905 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 07:10:28 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:28 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:28.830574 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:28.823035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:39629)
0620 07:10:28.823573 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:28.823626 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:28.823746 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:28.824017 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:28.824020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:28.824352 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:28.824506 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:28.824514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:28.825433 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:28.825437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:28.826306 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:28.826316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:28.826968 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:28.826972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:28.827101 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:28.827619 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:28.827635 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:28.830361 (+ 2726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":314,"threads_started":1}
W20260620 07:10:28.830770 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:28.823119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:39629 (local address 127.6.53.126:42251)
0620 07:10:28.823274 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:28.823278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:28.823703 (+ 425us) server_negotiation.cc:407] Connection header received
0620 07:10:28.823824 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:28.823828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:28.823886 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:28.823980 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:28.824633 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:28.825315 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:28.826416 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:28.826828 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:28.826936 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:28.830550 (+ 3614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:39629: BlockingRecv error: recv got EOF from 127.6.53.66:39629 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 07:10:29 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:29 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:29.256444 15611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:29.248527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:39435)
0620 07:10:29.249001 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:29.249021 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:29.249129 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:29.249407 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:29.249410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:29.249753 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:29.249902 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:29.249909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:29.250910 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:29.250914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:29.251793 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:29.251803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:29.252457 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:29.252461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:29.252576 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:29.253152 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:29.253168 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:29.256141 (+ 2973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":253,"threads_started":1}
W20260620 07:10:29.256544 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:29.248628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:39435 (local address 127.6.53.126:42251)
0620 07:10:29.248871 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:29.248877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:29.249113 (+ 236us) server_negotiation.cc:407] Connection header received
0620 07:10:29.249225 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:29.249228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:29.249289 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:29.249366 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:29.250017 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:29.250791 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:29.251909 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:29.252335 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:29.252423 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:29.256310 (+ 3887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:39435: BlockingRecv error: recv got EOF from 127.6.53.67:39435 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 07:10:29 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:29 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:29.389585 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:29.381482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:45869)
0620 07:10:29.382020 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:29.382036 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:29.382147 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:29.382495 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:29.382498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:29.382899 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:29.383055 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:29.383063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:29.384259 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:29.384265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:29.385188 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:29.385201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:29.385889 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:29.385895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:29.386062 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:29.386690 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:29.386708 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:29.389345 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":233,"threads_started":1}
W20260620 07:10:29.389739 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:29.381569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:45869 (local address 127.6.53.126:42251)
0620 07:10:29.381822 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:29.381831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:29.382120 (+ 289us) server_negotiation.cc:407] Connection header received
0620 07:10:29.382267 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:29.382271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:29.382332 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:29.382462 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:29.383181 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:29.384113 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:29.385340 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:29.385736 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:29.385870 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:29.389516 (+ 3646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:45869: BlockingRecv error: recv got EOF from 127.6.53.65:45869 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 07:10:29 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:29 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 07:10:29.846125 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:29.831533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:36225 (local address 127.6.53.126:42251)
0620 07:10:29.831755 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:29.831760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:29.832150 (+ 390us) server_negotiation.cc:407] Connection header received
0620 07:10:29.832276 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:29.832280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:29.832339 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:29.832418 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:29.833215 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:29.833942 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:29.835592 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:29.836283 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:29.836369 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:29.845859 (+ 9490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:36225: BlockingRecv error: recv got EOF from 127.6.53.66:36225 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260620 07:10:29.846756 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:29.831495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:36225)
0620 07:10:29.832059 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:29.832074 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:29.832188 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:29.832458 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:29.832461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:29.832920 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:29.833096 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:29.833105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:29.834058 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:29.834061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:29.835438 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:29.835452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:29.836597 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:29.836602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:29.836758 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:29.842151 (+ 5393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:29.842173 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:29.846406 (+ 4233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":246,"threads_started":1}
Jun 20 07:10:30 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:30 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:30.265695 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:30.257427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:54491)
0620 07:10:30.258052 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:30.258071 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:30.258227 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:30.258596 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:30.258601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:30.259069 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:30.259236 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:30.259246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:30.260242 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:30.260248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:30.261176 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:30.261187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:30.261947 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:30.261953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:30.262109 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:30.262711 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:30.262729 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:30.265452 (+ 2723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":281,"threads_started":1}
W20260620 07:10:30.265865 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:30.257528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:54491 (local address 127.6.53.126:42251)
0620 07:10:30.257772 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:30.257781 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:30.258152 (+ 371us) server_negotiation.cc:407] Connection header received
0620 07:10:30.258375 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:30.258380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:30.258455 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:30.258563 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:30.259381 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:30.260091 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:30.261321 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:30.261773 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:30.261911 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:30.265647 (+ 3736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:54491: BlockingRecv error: recv got EOF from 127.6.53.67:54491 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 07:10:30 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:30 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:30.398782 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:30.390495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:38585)
0620 07:10:30.391068 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:30.391085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:30.391256 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:30.391610 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:30.391615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:30.392015 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:30.392159 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:30.392168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:30.393159 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:30.393165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:30.394130 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:30.394140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:30.394885 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:30.394892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:30.395066 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:30.395645 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:30.395663 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:30.398443 (+ 2780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":307,"threads_started":1}
W20260620 07:10:30.398892 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:30.390714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:38585 (local address 127.6.53.126:42251)
0620 07:10:30.390960 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:30.390966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:30.391181 (+ 215us) server_negotiation.cc:407] Connection header received
0620 07:10:30.391394 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:30.391405 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:30.391475 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:30.391579 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:30.392295 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:30.393018 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:30.394294 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:30.394706 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:30.394842 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:30.398617 (+ 3775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:38585: BlockingRecv error: recv got EOF from 127.6.53.65:38585 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 07:10:30 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:30 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:30.856140 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:30.847807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:47143)
0620 07:10:30.848283 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:30.848298 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:30.848449 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:30.848776 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:30.848781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:30.849199 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:30.849335 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:30.849343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:30.850575 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:30.850582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:30.851485 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:30.851495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:30.852264 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:30.852271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:30.852425 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:30.853052 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:30.853069 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:30.855824 (+ 2755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":231,"threads_started":1}
W20260620 07:10:30.856235 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:30.847890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:47143 (local address 127.6.53.126:42251)
0620 07:10:30.848114 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:30.848122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:30.848382 (+ 260us) server_negotiation.cc:407] Connection header received
0620 07:10:30.848570 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:30.848576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:30.848649 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:30.848748 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:30.849477 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:30.850439 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:30.851635 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:30.852078 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:30.852228 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:30.856009 (+ 3781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:47143: BlockingRecv error: recv got EOF from 127.6.53.66:47143 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 07:10:31 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:31 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:31.275520 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:31.266730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:35541)
0620 07:10:31.267232 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:31.267251 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:31.267362 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:31.267640 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:31.267643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:31.267990 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:31.268127 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:31.268135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:31.269136 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:31.269140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:31.270024 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:31.270033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:31.270792 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:31.270796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:31.270902 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:31.271523 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:31.271548 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:31.275297 (+ 3749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":290,"threads_started":1}
W20260620 07:10:31.275748 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:31.266776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:35541 (local address 127.6.53.126:42251)
0620 07:10:31.267023 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:31.267033 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:31.267343 (+ 310us) server_negotiation.cc:407] Connection header received
0620 07:10:31.267454 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:31.267458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:31.267520 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:31.267603 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:31.268243 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:31.269023 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:31.270144 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:31.270633 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:31.270724 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:31.275493 (+ 4769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:35541: BlockingRecv error: recv got EOF from 127.6.53.67:35541 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 07:10:31 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:31 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:31.410365 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:31.399943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:54135)
0620 07:10:31.400861 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:31.400893 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:31.401051 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:31.401608 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:31.401616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:31.402183 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:31.402457 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:31.402472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:31.404064 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:31.404070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:31.405453 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:31.405466 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:31.406462 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:31.406469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:31.406646 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:31.407292 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:31.407310 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:31.409988 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":673,"threads_started":1}
W20260620 07:10:31.410528 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:31.400020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:54135 (local address 127.6.53.126:42251)
0620 07:10:31.400333 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:31.400342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:31.400998 (+ 656us) server_negotiation.cc:407] Connection header received
0620 07:10:31.401197 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:31.401204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:31.401298 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:31.401412 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:31.402636 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:31.403904 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:31.405596 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:31.406305 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:31.406433 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:31.410204 (+ 3771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:54135: BlockingRecv error: recv got EOF from 127.6.53.65:54135 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Jun 20 07:10:31 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:31 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:31.865463 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:31.857174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:33081)
0620 07:10:31.857714 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:31.857734 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:31.857872 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:31.858196 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:31.858201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:31.858734 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:31.858904 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:31.858913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:31.859848 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:31.859852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:31.860990 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:31.861002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:31.861720 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:31.861724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:31.861858 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:31.862455 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:31.862471 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:31.865236 (+ 2765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":283,"threads_started":1}
W20260620 07:10:31.865751 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:31.857256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:33081 (local address 127.6.53.126:42251)
0620 07:10:31.857481 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:31.857490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:31.857839 (+ 349us) server_negotiation.cc:407] Connection header received
0620 07:10:31.857974 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:31.857980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:31.858050 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:31.858127 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:31.859038 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:31.859741 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:31.861138 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:31.861592 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:31.861694 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:31.865453 (+ 3759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:33081: BlockingRecv error: recv got EOF from 127.6.53.66:33081 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 07:10:32 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:32 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:32.285105 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:32.276683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:39797)
0620 07:10:32.277239 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:32.277265 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:32.277398 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:32.277749 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:32.277754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:32.278298 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:32.278526 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:32.278538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:32.279516 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:32.279524 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:32.280471 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:32.280482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:32.281188 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:32.281194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:32.281374 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:32.282025 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:32.282043 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:32.284751 (+ 2708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":305,"threads_started":1}
W20260620 07:10:32.285137 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:32.276735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:39797 (local address 127.6.53.126:42251)
0620 07:10:32.276959 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:32.276966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:32.277327 (+ 361us) server_negotiation.cc:407] Connection header received
0620 07:10:32.277521 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:32.277527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:32.277622 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:32.277713 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:32.278689 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:32.279387 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:32.280610 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:32.281036 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:32.281171 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:32.284937 (+ 3766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:39797: BlockingRecv error: recv got EOF from 127.6.53.67:39797 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 07:10:32 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:32 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:32.419796 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:32.411361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:46759)
0620 07:10:32.411888 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:32.411905 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:32.412060 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:32.412432 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:32.412437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:32.412823 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:32.412960 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:32.412968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:32.413992 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:32.413999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:32.415227 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:32.415243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:32.416100 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:32.416105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:32.416210 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:32.416876 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:32.416894 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:32.419573 (+ 2679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":296,"threads_started":1}
W20260620 07:10:32.419987 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:32.411524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:46759 (local address 127.6.53.126:42251)
0620 07:10:32.411757 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:32.411764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:32.411995 (+ 231us) server_negotiation.cc:407] Connection header received
0620 07:10:32.412201 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:32.412207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:32.412292 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:32.412400 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:32.413105 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:32.413852 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:32.415388 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:32.415961 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:32.416072 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:32.419757 (+ 3685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:46759: BlockingRecv error: recv got EOF from 127.6.53.65:46759 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 07:10:32 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:32 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:32.874382 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:32.866424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:47175)
0620 07:10:32.866944 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:32.866960 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:32.867130 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:32.867464 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:32.867467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:32.867861 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:32.867997 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:32.868005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:32.868991 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:32.868995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:32.869885 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:32.869894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:32.870628 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:32.870632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:32.870738 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:32.871318 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:32.871335 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:32.874077 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":303,"threads_started":1}
W20260620 07:10:32.874523 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:32.866454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:47175 (local address 127.6.53.126:42251)
0620 07:10:32.866727 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:32.866737 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:32.867063 (+ 326us) server_negotiation.cc:407] Connection header received
0620 07:10:32.867260 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:32.867266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:32.867340 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:32.867423 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:32.868119 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:32.868849 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:32.870006 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:32.870499 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:32.870594 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:32.874240 (+ 3646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:47175: BlockingRecv error: recv got EOF from 127.6.53.66:47175 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Jun 20 07:10:33 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:33 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:33.294477 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:33.286064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:60783)
0620 07:10:33.286649 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:33.286669 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:33.286817 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:33.287165 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:33.287170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:33.287558 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:33.287698 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:33.287706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:33.288696 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:33.288702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:33.289729 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:33.289741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:33.290517 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:33.290523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:33.290680 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:33.291307 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:33.291324 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:33.294113 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":373,"threads_started":1}
W20260620 07:10:33.294623 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:33.286284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:60783 (local address 127.6.53.126:42251)
0620 07:10:33.286505 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:33.286511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:33.286749 (+ 238us) server_negotiation.cc:407] Connection header received
0620 07:10:33.286941 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:33.286946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:33.287034 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:33.287134 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:33.287836 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:33.288558 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:33.289867 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:33.290360 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:33.290488 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:33.294363 (+ 3875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:60783: BlockingRecv error: recv got EOF from 127.6.53.67:60783 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 07:10:33 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:33 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:33.429162 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:33.420806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:57759)
0620 07:10:33.421318 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:33.421334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:33.421442 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:33.421745 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:33.421748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:33.422157 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:33.422357 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:33.422366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:33.423372 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:33.423376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:33.424247 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:33.424258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:33.425028 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:33.425034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:33.425212 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:33.425879 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:33.425901 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:33.428919 (+ 3018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":296,"threads_started":1}
W20260620 07:10:33.429301 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:33.420906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:57759 (local address 127.6.53.126:42251)
0620 07:10:33.421097 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:33.421101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:33.421406 (+ 305us) server_negotiation.cc:407] Connection header received
0620 07:10:33.421545 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:33.421548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:33.421620 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:33.421702 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:33.422470 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:33.423252 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:33.424399 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:33.424859 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:33.424953 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:33.429102 (+ 4149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:57759: BlockingRecv error: recv got EOF from 127.6.53.65:57759 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 07:10:33 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:33 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:33.883279 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:33.875271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:46657)
0620 07:10:33.875781 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:33.875797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:33.875909 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:33.876203 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:33.876206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:33.876582 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:33.876747 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:33.876755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:33.877703 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:33.877706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:33.878650 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:33.878663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:33.879309 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:33.879313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:33.879429 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:33.879991 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:33.880016 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:33.883054 (+ 3038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":278,"threads_started":1}
W20260620 07:10:33.883471 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:33.875554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:46657 (local address 127.6.53.126:42251)
0620 07:10:33.875758 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:33.875763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:33.875867 (+ 104us) server_negotiation.cc:407] Connection header received
0620 07:10:33.876017 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:33.876021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:33.876079 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:33.876156 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:33.876854 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:33.877588 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:33.878767 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:33.879168 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:33.879276 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:33.883222 (+ 3946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:46657: BlockingRecv error: recv got EOF from 127.6.53.66:46657 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 07:10:34 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:34 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:34.303923 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:34.295351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:38939)
0620 07:10:34.295809 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:34.295829 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:34.295988 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:34.296331 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:34.296336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:34.296841 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:34.297068 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:34.297081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:34.298332 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:34.298335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:34.299172 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:34.299183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:34.300069 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:34.300073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:34.300190 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:34.300776 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:34.300793 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:34.303686 (+ 2893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":259,"threads_started":1}
W20260620 07:10:34.304342 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:34.295410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:38939 (local address 127.6.53.126:42251)
0620 07:10:34.295633 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:34.295640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:34.295921 (+ 281us) server_negotiation.cc:407] Connection header received
0620 07:10:34.296126 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:34.296137 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:34.296199 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:34.296302 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:34.297206 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:34.298160 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:34.299344 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:34.299937 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:34.300026 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:34.303919 (+ 3893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:38939: BlockingRecv error: recv got EOF from 127.6.53.67:38939 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 07:10:34 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:34 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:34.438540 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:34.430156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:48533)
0620 07:10:34.430670 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:34.430685 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:34.430836 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:34.431246 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:34.431253 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:34.431671 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:34.431819 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:34.431828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:34.432822 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:34.432828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:34.433829 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:34.433841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:34.434717 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:34.434723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:34.434900 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:34.435550 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:34.435568 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:34.438300 (+ 2732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":303,"threads_started":1}
W20260620 07:10:34.438756 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:34.430186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:48533 (local address 127.6.53.126:42251)
0620 07:10:34.430412 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:34.430422 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:34.430766 (+ 344us) server_negotiation.cc:407] Connection header received
0620 07:10:34.430979 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:34.430984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:34.431097 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:34.431202 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:34.431985 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:34.432676 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:34.433978 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:34.434575 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:34.434710 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:34.438481 (+ 3771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:48533: BlockingRecv error: recv got EOF from 127.6.53.65:48533 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 07:10:34 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:34 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:34.897413 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:34.884296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:36187)
0620 07:10:34.884851 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:34.884867 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:34.884984 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:34.885277 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:34.885280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:34.885629 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:34.885776 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:34.885783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:34.886736 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:34.886740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:34.887623 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:34.887633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:34.888730 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:34.888734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:34.888943 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:34.894263 (+ 5320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:34.894282 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:34.897112 (+ 2830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":308,"threads_started":1}
W20260620 07:10:34.897564 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:34.884478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:36187 (local address 127.6.53.126:42251)
0620 07:10:34.884656 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:34.884660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:34.884945 (+ 285us) server_negotiation.cc:407] Connection header received
0620 07:10:34.885081 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:34.885085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:34.885142 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:34.885224 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:34.885895 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:34.886618 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:34.887805 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:34.888588 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:34.888728 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:34.897360 (+ 8632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:36187: BlockingRecv error: recv got EOF from 127.6.53.66:36187 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 07:10:35 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:35 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:35.313139 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:35.304974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:60215)
0620 07:10:35.305465 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:35.305484 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:35.305639 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:35.306021 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:35.306026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:35.306524 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:35.306678 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:35.306686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:35.307717 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:35.307724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:35.308627 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:35.308639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:35.309343 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:35.309348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:35.309530 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:35.310177 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:35.310195 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:35.312893 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":267,"threads_started":1}
W20260620 07:10:35.313323 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:35.305024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:60215 (local address 127.6.53.126:42251)
0620 07:10:35.305246 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:35.305254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:35.305572 (+ 318us) server_negotiation.cc:407] Connection header received
0620 07:10:35.305760 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:35.305765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:35.305865 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:35.305985 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:35.306826 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:35.307567 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:35.308773 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:35.309194 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:35.309333 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:35.313081 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:60215: BlockingRecv error: recv got EOF from 127.6.53.67:60215 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 07:10:35 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:35 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:35.447794 15630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:35.439378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:35221)
0620 07:10:35.439933 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:35.439949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:35.440066 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:35.440365 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:35.440368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:35.440727 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:35.440879 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:35.440887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:35.441823 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:35.441827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:35.442834 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:35.442846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:35.443553 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:35.443559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:35.443680 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:35.444269 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:35.444288 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:35.447563 (+ 3275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":318,"threads_started":1}
W20260620 07:10:35.447885 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:35.439492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:35221 (local address 127.6.53.126:42251)
0620 07:10:35.439653 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:35.439657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:35.440026 (+ 369us) server_negotiation.cc:407] Connection header received
0620 07:10:35.440161 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:35.440164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:35.440238 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:35.440319 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:35.441006 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:35.441700 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:35.442967 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:35.443368 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:35.443464 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:35.447709 (+ 4245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:35221: BlockingRecv error: recv got EOF from 127.6.53.65:35221 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 07:10:35 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:35 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:35.907073 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:35.898404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:46003)
0620 07:10:35.898949 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:35.898966 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:35.899078 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:35.899408 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:35.899411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:35.899787 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:35.899935 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:35.899943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:35.900987 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:35.900991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:35.902329 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:35.902344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:35.903272 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:35.903276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:35.903389 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:35.903954 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:35.903973 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:35.906832 (+ 2859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":285,"threads_started":1}
W20260620 07:10:35.907274 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:35.898528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:46003 (local address 127.6.53.126:42251)
0620 07:10:35.898688 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:35.898693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:35.899037 (+ 344us) server_negotiation.cc:407] Connection header received
0620 07:10:35.899172 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:35.899176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:35.899265 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:35.899403 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:35.900057 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:35.900834 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:35.902485 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:35.903095 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:35.903201 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:35.907032 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:46003: BlockingRecv error: recv got EOF from 127.6.53.66:46003 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 07:10:36 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:36 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:36.322404 15632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:36.314034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:44883)
0620 07:10:36.314618 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:36.314637 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:36.314826 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:36.315183 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:36.315187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:36.315597 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:36.315748 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:36.315756 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:36.316844 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:36.316852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:36.317797 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:36.317808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:36.318612 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:36.318618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:36.318774 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:36.319325 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:36.319343 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:36.322047 (+ 2704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":315,"threads_started":1}
W20260620 07:10:36.322532 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:36.314116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:44883 (local address 127.6.53.126:42251)
0620 07:10:36.314380 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:36.314387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:36.314746 (+ 359us) server_negotiation.cc:407] Connection header received
0620 07:10:36.314983 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:36.314988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:36.315048 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:36.315154 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:36.315892 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:36.316710 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:36.317950 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:36.318422 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:36.318551 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:36.322277 (+ 3726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:44883: BlockingRecv error: recv got EOF from 127.6.53.67:44883 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 20 07:10:36 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:36 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:36.457068 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:36.448708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:56735)
0620 07:10:36.449265 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:36.449283 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:36.449451 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:36.449842 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:36.449847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:36.450219 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:36.450455 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:36.450465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:36.451491 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:36.451496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:36.452379 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:36.452390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:36.453071 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:36.453076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:36.453245 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:36.453862 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:36.453879 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:36.456690 (+ 2811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":310,"threads_started":1}
W20260620 07:10:36.457134 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:36.448830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:56735 (local address 127.6.53.126:42251)
0620 07:10:36.449191 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:36.449197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:36.449373 (+ 176us) server_negotiation.cc:407] Connection header received
0620 07:10:36.449588 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:36.449598 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:36.449670 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:36.449777 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:36.450597 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:36.451348 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:36.452530 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:36.452929 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:36.453061 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:36.456895 (+ 3834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:56735: BlockingRecv error: recv got EOF from 127.6.53.65:56735 (error 108)
Metrics: {"server-negotiator.queue_time_us":225}
Jun 20 07:10:36 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:36 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:36.916607 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:36.908022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:51491)
0620 07:10:36.908537 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:36.908554 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:36.908691 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:36.908983 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:36.908986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:36.909391 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:36.909582 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:36.909591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:36.910673 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:36.910677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:36.911584 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:36.911595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:36.912288 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:36.912292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:36.912404 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:36.913041 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:36.913058 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:36.916244 (+ 3186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":271,"threads_started":1}
W20260620 07:10:36.916636 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:36.908199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:51491 (local address 127.6.53.126:42251)
0620 07:10:36.908437 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:36.908443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:36.908636 (+ 193us) server_negotiation.cc:407] Connection header received
0620 07:10:36.908792 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:36.908796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:36.908857 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:36.908938 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:36.909714 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:36.910527 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:36.911715 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:36.912136 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:36.912273 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:36.916427 (+ 4154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:51491: BlockingRecv error: recv got EOF from 127.6.53.66:51491 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Jun 20 07:10:37 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:37 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.67@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:37.331349 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:37.323387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.67:40205)
0620 07:10:37.323864 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:37.323884 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:37.324038 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:37.324368 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:37.324373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:37.324758 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:37.324928 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:37.324937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:37.325925 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:37.325933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:37.326857 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:37.326868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:37.327534 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:37.327540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:37.327720 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:37.328283 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:37.328299 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:37.330980 (+ 2681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":263,"threads_started":1}
W20260620 07:10:37.331437 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:37.323443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:40205 (local address 127.6.53.126:42251)
0620 07:10:37.323682 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:37.323690 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:37.323972 (+ 282us) server_negotiation.cc:407] Connection header received
0620 07:10:37.324159 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:37.324165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:37.324233 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:37.324336 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:37.325078 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:37.325774 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:37.327009 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:37.327387 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:37.327522 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:37.331204 (+ 3682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.67:40205: BlockingRecv error: recv got EOF from 127.6.53.67:40205 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 07:10:37 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:37 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.65@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:37.466063 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:37.458199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.65:59371)
0620 07:10:37.458767 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:37.458783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:37.458889 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:37.459184 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:37.459187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:37.459576 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:37.459757 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:37.459765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:37.460690 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:37.460693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:37.461587 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:37.461598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:37.462234 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:37.462238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:37.462415 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:37.462983 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:37.463000 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:37.465749 (+ 2749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":341,"threads_started":1}
W20260620 07:10:37.466224 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:37.458237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:59371 (local address 127.6.53.126:42251)
0620 07:10:37.458435 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:37.458439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:37.458854 (+ 415us) server_negotiation.cc:407] Connection header received
0620 07:10:37.458987 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:37.458991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:37.459064 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:37.459142 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:37.459884 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:37.460571 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:37.461709 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:37.462116 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:37.462206 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:37.465987 (+ 3781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.65:59371: BlockingRecv error: recv got EOF from 127.6.53.65:59371 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 07:10:37 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:10:37 dist-test-slave-4n6h krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.53.66@KRBTEST.COM for oryx/127.6.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:10:37.925573 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:37.917673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42251 (local address 127.6.53.66:35807)
0620 07:10:37.918204 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:37.918227 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:37.918355 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:37.918642 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:37.918645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:37.919043 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:37.919221 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:37.919230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:37.920257 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:37.920263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:37.921216 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:37.921228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:37.921901 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:37.921905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:37.922010 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:37.922590 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:37.922609 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:37.925312 (+ 2703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.53.126:42251: Server oryx/127.6.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":312,"threads_started":1}
W20260620 07:10:37.925752 15329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:37.917762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:35807 (local address 127.6.53.126:42251)
0620 07:10:37.917919 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:37.917923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:37.918301 (+ 378us) server_negotiation.cc:407] Connection header received
0620 07:10:37.918452 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:37.918455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:37.918515 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:37.918593 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:37.919338 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:37.920103 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:37.921335 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:37.921776 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:37.921865 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:37.925507 (+ 3642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.53.66:35807: BlockingRecv error: recv got EOF from 127.6.53.66:35807 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:10:38.177107 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 15179
I20260620 07:10:38.189404 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:38.178129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.6.53.65:41005)
0620 07:10:38.178910 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:38.178917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:38.178940 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:10:38.178992 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:38.178998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:38.179213 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:38.179408 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:38.180842 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:38.181835 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:38.183134 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:38.183665 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:38.183811 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:38.186221 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:38.186228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:38.186241 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:38.186327 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:38.188753 (+ 2426us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.53.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 07:10:38.189080 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38092: Request ticket server kudu/127.6.53.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":535,"threads_started":1}
W20260620 07:10:38.189478 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 15179: Not authorized: Client connection negotiation failed: client connection to 127.6.53.65:41005: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.53.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 07:10:38.189646 15639 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38092: Request ticket server kudu/127.6.53.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 07:10:38.189711 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 15179
I20260620 07:10:38.212386 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 15330
I20260620 07:10:38.227828 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:38.213397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45218 (local address 127.6.53.66:34915)
0620 07:10:38.214187 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:38.214194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:38.214225 (+ 31us) server_negotiation.cc:407] Connection header received
0620 07:10:38.214330 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:38.214337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:38.214551 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:38.214694 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:38.216741 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:38.217999 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:38.220121 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:38.220923 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:38.221064 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:38.224735 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:38.224745 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:38.224758 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:38.224791 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:38.227337 (+ 2546us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.53.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 07:10:38.227600 (+ 263us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45218: Request ticket server kudu/127.6.53.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":433,"threads_started":1}
W20260620 07:10:38.227947 15640 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45218: Request ticket server kudu/127.6.53.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 07:10:38.228330 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 15330: Not authorized: Client connection negotiation failed: client connection to 127.6.53.66:34915: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.53.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 07:10:38.228493 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 15330
I20260620 07:10:38.247108 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 15461
I20260620 07:10:38.258692 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:38.248106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.6.53.67:39353)
0620 07:10:38.248727 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:38.248735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:38.248784 (+ 49us) server_negotiation.cc:407] Connection header received
0620 07:10:38.248841 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:38.248847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:38.249093 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:38.249239 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:38.250505 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:38.251618 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:38.252692 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:38.253204 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:38.253298 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:38.255703 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:38.255711 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:38.255724 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:38.255764 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:38.258047 (+ 2283us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.53.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 07:10:38.258399 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58948: Request ticket server kudu/127.6.53.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":351,"threads_started":1}
W20260620 07:10:38.258801 15641 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58948: Request ticket server kudu/127.6.53.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 07:10:38.258800 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 15461: Not authorized: Client connection negotiation failed: client connection to 127.6.53.67:39353: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.53.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 07:10:38.259003 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 15461
I20260620 07:10:38.275175 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 15110
I20260620 07:10:38.276397 15172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:38.391232 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 15110
2026-06-20T07:10:38Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20912 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T07:10:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:10:38Z Disabled control of system clock
I20260620 07:10:38.466583 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:40853
--webserver_interface=127.6.53.126
--webserver_port=0
--builtin_ntp_servers=127.6.53.84:34629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:40853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 07:10:38.779459 15646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:38.779719 15646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:38.779795 15646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:38.786913 15646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:10:38.787014 15646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:38.787094 15646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:38.787211 15646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:10:38.787292 15646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:10:38.796317 15646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:40853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:40853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:38.797609 15646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:38.799319 15646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:38.811344 15651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:38.811301 15652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:38.811319 15654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:38.811895 15646 server_base.cc:1034] running on GCE node
I20260620 07:10:38.812878 15646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:38.814327 15646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:38.815555 15646 hybrid_clock.cc:648] HybridClock initialized: now 1781939438815503 us; error 46 us; skew 500 ppm
I20260620 07:10:38.815889 15646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:10:38.818006 15646 webserver.cc:460] Webserver started at http://127.6.53.126:33517/ using document root <none> and password file <none>
I20260620 07:10:38.818718 15646 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:38.818856 15646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:38.819168 15646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:38.821900 15646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "d53bfc8bd2794760b08257da4fd6ee4e"
format_stamp: "Formatted at 2026-06-20 07:10:38 on dist-test-slave-4n6h"
server_key: "1218be787dc1e5b82203f3de8b7749cc"
server_key_iv: "69c9d64014bd9098008ed7490b9132af"
server_key_version: "encryptionkey@0"
I20260620 07:10:38.822672 15646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "d53bfc8bd2794760b08257da4fd6ee4e"
format_stamp: "Formatted at 2026-06-20 07:10:38 on dist-test-slave-4n6h"
server_key: "1218be787dc1e5b82203f3de8b7749cc"
server_key_iv: "69c9d64014bd9098008ed7490b9132af"
server_key_version: "encryptionkey@0"
I20260620 07:10:38.827500 15646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 07:10:38.830931 15660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:38.832199 15646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:10:38.832396 15646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "d53bfc8bd2794760b08257da4fd6ee4e"
format_stamp: "Formatted at 2026-06-20 07:10:38 on dist-test-slave-4n6h"
server_key: "1218be787dc1e5b82203f3de8b7749cc"
server_key_iv: "69c9d64014bd9098008ed7490b9132af"
server_key_version: "encryptionkey@0"
I20260620 07:10:38.832630 15646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:38.864581 15646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:38.865490 15646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:38.865797 15646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:38.884153 15646 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:40853
I20260620 07:10:38.884161 15711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:40853 every 8 connection(s)
I20260620 07:10:38.885515 15646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:10:38.889063 15712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:38.895777 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 15646
I20260620 07:10:38.895988 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:10:38.896766 6357 external_mini_cluster.cc:1428] Setting key 3832945257ebcf920829d9f4a15d63e6
I20260620 07:10:38.900333 15712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e: Bootstrap starting.
I20260620 07:10:38.903388 15712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:38.904402 15712 log.cc:824] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:38.907131 15712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e: No bootstrap required, opened a new log
I20260620 07:10:38.910306 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:38.898879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.6.53.126:40853)
0620 07:10:38.899833 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:38.899844 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:38.899883 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:10:38.901600 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:38.901625 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:38.902064 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:38.902494 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:38.903102 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:38.904467 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:38.905807 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:38.906544 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:38.906748 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:38.907980 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:38.907988 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:38.908001 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:10:38.908538 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:38.908621 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:10:38.908642 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:38.908784 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:38.909004 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 07:10:38.909321 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":351,"threads_started":1}
I20260620 07:10:38.915704 15712 raft_consensus.cc:348] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53bfc8bd2794760b08257da4fd6ee4e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40853 } }
I20260620 07:10:38.916191 15712 raft_consensus.cc:374] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:38.916328 15712 raft_consensus.cc:729] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d53bfc8bd2794760b08257da4fd6ee4e, State: Initialized, Role: FOLLOWER
I20260620 07:10:38.916919 15712 consensus_queue.cc:260] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53bfc8bd2794760b08257da4fd6ee4e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40853 } }
I20260620 07:10:38.917162 15712 raft_consensus.cc:388] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:10:38.917310 15712 raft_consensus.cc:482] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:10:38.917498 15712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:38.919631 15712 raft_consensus.cc:504] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53bfc8bd2794760b08257da4fd6ee4e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40853 } }
I20260620 07:10:38.920079 15712 leader_election.cc:302] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d53bfc8bd2794760b08257da4fd6ee4e; no voters:
I20260620 07:10:38.920639 15712 leader_election.cc:288] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:10:38.920768 15717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:10:38.921463 15717 raft_consensus.cc:686] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [term 1 LEADER]: Becoming Leader. State: Replica: d53bfc8bd2794760b08257da4fd6ee4e, State: Running, Role: LEADER
I20260620 07:10:38.922000 15717 consensus_queue.cc:237] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53bfc8bd2794760b08257da4fd6ee4e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40853 } }
I20260620 07:10:38.922693 15712 sys_catalog.cc:564] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:10:38.925235 15718 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d53bfc8bd2794760b08257da4fd6ee4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53bfc8bd2794760b08257da4fd6ee4e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40853 } } }
I20260620 07:10:38.925235 15719 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d53bfc8bd2794760b08257da4fd6ee4e. Latest consensus state: current_term: 1 leader_uuid: "d53bfc8bd2794760b08257da4fd6ee4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53bfc8bd2794760b08257da4fd6ee4e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40853 } } }
I20260620 07:10:38.925683 15719 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:38.925683 15718 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:38.930650 15723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:10:38.935463 15723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:10:38.944162 15723 catalog_manager.cc:1347] Generated new cluster ID: 57532ecc59c4437aa3a422b0d1290226
I20260620 07:10:38.944278 15723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:10:38.970795 15723 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:10:38.972085 15723 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:10:38.982030 15723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d53bfc8bd2794760b08257da4fd6ee4e: Generated new TSK 0
I20260620 07:10:38.983016 15723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:10:38.995612 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40853
--builtin_ntp_servers=127.6.53.84:34629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 07:10:39.319731 15736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:39.320032 15736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:39.320191 15736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:39.327708 15736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:39.327832 15736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:39.327996 15736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:10:39.337448 15736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:39.338773 15736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:39.340528 15736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:39.353281 15741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:39.353232 15742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:39.353262 15744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:39.354108 15736 server_base.cc:1034] running on GCE node
I20260620 07:10:39.354856 15736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:39.355885 15736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:39.357082 15736 hybrid_clock.cc:648] HybridClock initialized: now 1781939439357016 us; error 59 us; skew 500 ppm
I20260620 07:10:39.357409 15736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:10:39.359575 15736 webserver.cc:460] Webserver started at http://127.6.53.65:34669/ using document root <none> and password file <none>
I20260620 07:10:39.360374 15736 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:39.360489 15736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:39.360823 15736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:39.363683 15736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "e3577c53dc0d461ca5bd06e65fe9be82"
format_stamp: "Formatted at 2026-06-20 07:10:39 on dist-test-slave-4n6h"
server_key: "bcb0d06be038edea40d28aeea7bf6aba"
server_key_iv: "46d84686d5e3619bcf24658639c029b2"
server_key_version: "encryptionkey@0"
I20260620 07:10:39.364467 15736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "e3577c53dc0d461ca5bd06e65fe9be82"
format_stamp: "Formatted at 2026-06-20 07:10:39 on dist-test-slave-4n6h"
server_key: "bcb0d06be038edea40d28aeea7bf6aba"
server_key_iv: "46d84686d5e3619bcf24658639c029b2"
server_key_version: "encryptionkey@0"
I20260620 07:10:39.369585 15736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 07:10:39.372985 15750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:39.374310 15736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:10:39.374467 15736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "e3577c53dc0d461ca5bd06e65fe9be82"
format_stamp: "Formatted at 2026-06-20 07:10:39 on dist-test-slave-4n6h"
server_key: "bcb0d06be038edea40d28aeea7bf6aba"
server_key_iv: "46d84686d5e3619bcf24658639c029b2"
server_key_version: "encryptionkey@0"
I20260620 07:10:39.374648 15736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:39.401240 15736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:39.402052 15736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:39.402323 15736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:39.403251 15736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:39.404677 15736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:39.404767 15736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:39.404850 15736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:39.404909 15736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:39.437960 15736 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:39505
I20260620 07:10:39.438046 15862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:39505 every 8 connection(s)
I20260620 07:10:39.439214 15736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:10:39.447930 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 15736
I20260620 07:10:39.448122 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:10:39.448745 6357 external_mini_cluster.cc:1428] Setting key 969afa41ca12c7c06af8a0c48d954090
I20260620 07:10:39.451541 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40853
--builtin_ntp_servers=127.6.53.84:34629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 07:10:39.451993 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:39.441662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:58985 (local address 127.6.53.126:40853)
0620 07:10:39.442095 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:39.442100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:39.443311 (+ 1211us) server_negotiation.cc:407] Connection header received
0620 07:10:39.445114 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:39.445119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:39.445212 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:39.445334 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:39.446472 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:39.447228 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:39.448959 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:39.449395 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:39.449540 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:39.451132 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:39.451137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:39.451140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:10:39.451157 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:39.451207 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:10:39.451218 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:39.451313 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:39.451559 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 07:10:39.451652 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":244,"threads_started":1}
I20260620 07:10:39.452962 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:39.442238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:40853 (local address 127.6.53.65:58985)
0620 07:10:39.443153 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:39.443186 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:39.444829 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:39.445571 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:39.445580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:39.445684 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:39.446221 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:39.446237 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:39.447400 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:39.447410 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:39.448771 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:39.448812 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:39.449637 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:39.449647 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:39.449937 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:39.450775 (+ 838us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:10:39.450781 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:39.450840 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:10:39.450937 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:39.451324 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:39.451345 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:39.451355 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 07:10:39.451541 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 07:10:39.451926 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":234,"threads_started":1}
I20260620 07:10:39.454478 15863 heartbeater.cc:347] Connected to a master server at 127.6.53.126:40853
I20260620 07:10:39.454783 15863 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:39.455510 15863 heartbeater.cc:511] Master 127.6.53.126:40853 requested a full tablet report, sending...
I20260620 07:10:39.457499 15677 ts_manager.cc:194] Registered new tserver with Master: e3577c53dc0d461ca5bd06e65fe9be82 (127.6.53.65:39505)
I20260620 07:10:39.459004 15677 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.53.65:58985
W20260620 07:10:39.768386 15867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:39.768649 15867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:39.768733 15867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:39.775974 15867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:39.776073 15867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:39.776247 15867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:10:39.785432 15867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:39.786731 15867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:39.788426 15867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:39.801591 15872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:39.801689 15873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:39.801790 15875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:39.802927 15867 server_base.cc:1034] running on GCE node
I20260620 07:10:39.803484 15867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:39.804332 15867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:39.805531 15867 hybrid_clock.cc:648] HybridClock initialized: now 1781939439805478 us; error 42 us; skew 500 ppm
I20260620 07:10:39.805852 15867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:10:39.808161 15867 webserver.cc:460] Webserver started at http://127.6.53.66:43741/ using document root <none> and password file <none>
I20260620 07:10:39.808841 15867 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:39.808952 15867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:39.809283 15867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:39.812031 15867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "f8428fe7ccc249deb4261d1a0f1963a6"
format_stamp: "Formatted at 2026-06-20 07:10:39 on dist-test-slave-4n6h"
server_key: "9572fe563ea3689aba53fd9128429350"
server_key_iv: "b662606ab213316f22ce0fd8720303cc"
server_key_version: "encryptionkey@0"
I20260620 07:10:39.812817 15867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "f8428fe7ccc249deb4261d1a0f1963a6"
format_stamp: "Formatted at 2026-06-20 07:10:39 on dist-test-slave-4n6h"
server_key: "9572fe563ea3689aba53fd9128429350"
server_key_iv: "b662606ab213316f22ce0fd8720303cc"
server_key_version: "encryptionkey@0"
I20260620 07:10:39.817729 15867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:10:39.820945 15881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:39.822185 15867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:10:39.822387 15867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "f8428fe7ccc249deb4261d1a0f1963a6"
format_stamp: "Formatted at 2026-06-20 07:10:39 on dist-test-slave-4n6h"
server_key: "9572fe563ea3689aba53fd9128429350"
server_key_iv: "b662606ab213316f22ce0fd8720303cc"
server_key_version: "encryptionkey@0"
I20260620 07:10:39.822594 15867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:39.856215 15867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:39.857075 15867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:39.857363 15867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:39.858481 15867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:39.859927 15867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:39.860016 15867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:39.860176 15867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:39.860266 15867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:39.893229 15867 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:34991
I20260620 07:10:39.893276 15993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:34991 every 8 connection(s)
I20260620 07:10:39.894670 15867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:10:39.903541 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 15867
I20260620 07:10:39.903707 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:10:39.904277 6357 external_mini_cluster.cc:1428] Setting key bf58d47c148942b09079d7bb0268b97a
I20260620 07:10:39.906922 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40853
--builtin_ntp_servers=127.6.53.84:34629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 07:10:39.916581 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:39.897163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:56193 (local address 127.6.53.126:40853)
0620 07:10:39.897388 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:39.897395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:39.898880 (+ 1485us) server_negotiation.cc:407] Connection header received
0620 07:10:39.900548 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:39.900554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:39.900640 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:39.900761 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:39.901700 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:39.902833 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:39.905040 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:39.905439 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:39.905578 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:39.915785 (+ 10207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:39.915790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:39.915794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:10:39.915811 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:39.915842 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:10:39.915853 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:39.915956 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:39.916177 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 07:10:39.916292 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 07:10:39.917660 15996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:39.897684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:40853 (local address 127.6.53.66:56193)
0620 07:10:39.898713 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:39.898760 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:39.900285 (+ 1525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:39.900934 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:39.900942 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:39.901018 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:39.901548 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:39.901563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:39.902982 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:39.902985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:39.904904 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:39.904916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:39.905672 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:39.905680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:39.905934 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:39.915416 (+ 9482us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:10:39.915419 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:39.915463 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:10:39.915553 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:39.915954 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:39.915967 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:39.915978 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 07:10:39.916170 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 07:10:39.916585 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":342,"threads_started":1}
I20260620 07:10:39.919225 15994 heartbeater.cc:347] Connected to a master server at 127.6.53.126:40853
I20260620 07:10:39.919530 15994 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:39.920253 15994 heartbeater.cc:511] Master 127.6.53.126:40853 requested a full tablet report, sending...
I20260620 07:10:39.921645 15677 ts_manager.cc:194] Registered new tserver with Master: f8428fe7ccc249deb4261d1a0f1963a6 (127.6.53.66:34991)
I20260620 07:10:39.922597 15677 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.53.66:56193
W20260620 07:10:40.226938 15997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:40.227236 15997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:40.227394 15997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:40.234812 15997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:40.234942 15997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:40.235097 15997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:10:40.243896 15997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:40.245146 15997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:40.246929 15997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:40.259697 16005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:40.259750 16002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:40.259871 16003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:40.260789 15997 server_base.cc:1034] running on GCE node
I20260620 07:10:40.261415 15997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:40.262399 15997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:40.263615 15997 hybrid_clock.cc:648] HybridClock initialized: now 1781939440263557 us; error 53 us; skew 500 ppm
I20260620 07:10:40.263926 15997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:10:40.266032 15997 webserver.cc:460] Webserver started at http://127.6.53.67:32867/ using document root <none> and password file <none>
I20260620 07:10:40.266762 15997 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:40.266889 15997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:40.267221 15997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:40.269946 15997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "451d27d526bc43e4990a7cc52a1717aa"
format_stamp: "Formatted at 2026-06-20 07:10:40 on dist-test-slave-4n6h"
server_key: "a59b5549da8f1002df4d9d1172531ffe"
server_key_iv: "1eff65d7784ff8ce40a42ca71ccc3750"
server_key_version: "encryptionkey@0"
I20260620 07:10:40.270761 15997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "451d27d526bc43e4990a7cc52a1717aa"
format_stamp: "Formatted at 2026-06-20 07:10:40 on dist-test-slave-4n6h"
server_key: "a59b5549da8f1002df4d9d1172531ffe"
server_key_iv: "1eff65d7784ff8ce40a42ca71ccc3750"
server_key_version: "encryptionkey@0"
I20260620 07:10:40.275591 15997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 07:10:40.278852 16011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:40.280052 15997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:10:40.280220 15997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "451d27d526bc43e4990a7cc52a1717aa"
format_stamp: "Formatted at 2026-06-20 07:10:40 on dist-test-slave-4n6h"
server_key: "a59b5549da8f1002df4d9d1172531ffe"
server_key_iv: "1eff65d7784ff8ce40a42ca71ccc3750"
server_key_version: "encryptionkey@0"
I20260620 07:10:40.280436 15997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:40.316432 15997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:40.317272 15997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:40.317574 15997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:40.318559 15997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:40.319967 15997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:40.320092 15997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:40.320261 15997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:40.320356 15997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:40.353519 15997 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:38681
I20260620 07:10:40.353577 16123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:38681 every 8 connection(s)
I20260620 07:10:40.354856 15997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:10:40.358598 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 15997
I20260620 07:10:40.358767 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:10:40.359206 6357 external_mini_cluster.cc:1428] Setting key 8fb17f63f0a53a28f567b73b587935d4
I20260620 07:10:40.367772 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:40.357295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:59567 (local address 127.6.53.126:40853)
0620 07:10:40.357468 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:40.357473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:40.358902 (+ 1429us) server_negotiation.cc:407] Connection header received
0620 07:10:40.360911 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:40.360916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:40.361012 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:40.361120 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:40.362818 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:40.363544 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:40.365052 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:40.365497 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:40.365636 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:40.366968 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:40.366974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:40.366978 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:10:40.367001 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:40.367030 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:10:40.367041 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:40.367134 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:40.367357 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 07:10:40.367449 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:10:40.368780 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:40.357790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:40853 (local address 127.6.53.67:59567)
0620 07:10:40.358717 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:40.358765 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:40.360643 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:40.361955 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:40.361963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:40.362046 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:40.362657 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:40.362674 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:40.363696 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:40.363706 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:40.364905 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:40.364917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:40.365721 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:40.365729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:40.365958 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:40.366618 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:10:40.366621 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:40.366656 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:10:40.366745 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:40.367133 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:40.367144 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:40.367154 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 07:10:40.367326 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 07:10:40.367690 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":261,"threads_started":1}
I20260620 07:10:40.370159 16124 heartbeater.cc:347] Connected to a master server at 127.6.53.126:40853
I20260620 07:10:40.370471 16124 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:40.371205 16124 heartbeater.cc:511] Master 127.6.53.126:40853 requested a full tablet report, sending...
I20260620 07:10:40.372560 15677 ts_manager.cc:194] Registered new tserver with Master: 451d27d526bc43e4990a7cc52a1717aa (127.6.53.67:38681)
I20260620 07:10:40.373461 15677 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.53.67:59567
I20260620 07:10:40.374320 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 07:10:40.380077 15865 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:40.378627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38844 (local address 127.6.53.126:40853)
0620 07:10:40.378842 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:40.378849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:40.379243 (+ 394us) server_negotiation.cc:407] Connection header received
0620 07:10:40.379413 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:40.379417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:40.379483 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:40.379555 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:40.379851 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38844: BlockingRecv error: recv got EOF from 127.0.0.1:38844 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:10:40.380535 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 15736
I20260620 07:10:40.387974 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:40.381410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51994 (local address 127.6.53.65:39505)
0620 07:10:40.381961 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:40.381968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:40.381992 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:10:40.382045 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:40.382050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:40.382233 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:40.382413 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:40.382699 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:40.383688 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:40.384951 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:40.385528 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:40.385621 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:40.386724 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:40.386733 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:40.386744 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:10:40.387313 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:40.387357 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:10:40.387375 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:40.387474 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:40.387579 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 07:10:40.387679 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":267,"threads_started":1}
I20260620 07:10:40.389132 15858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:40.470671 15863 heartbeater.cc:503] Master 127.6.53.126:40853 was elected leader, sending a full tablet report...
I20260620 07:10:40.471844 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 15736
W20260620 07:10:40.489141 15661 connection.cc:535] server connection from 127.6.53.65:58985 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 07:10:40.489552 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 15867
I20260620 07:10:40.497066 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:40.490546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.6.53.66:34991)
0620 07:10:40.491078 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:40.491085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:40.491109 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:10:40.491173 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:40.491178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:40.491402 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:40.491530 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:40.491891 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:40.492871 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:40.493990 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:40.494608 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:40.494705 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:40.495684 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:40.495692 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:40.495704 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:10:40.496280 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:40.496336 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:10:40.496366 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:40.496450 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:40.496572 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 07:10:40.496713 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":276,"threads_started":1}
I20260620 07:10:40.498386 15989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:40.581141 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 15867
I20260620 07:10:40.597235 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 15997
I20260620 07:10:40.605116 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:40.598577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.6.53.67:38681)
0620 07:10:40.599145 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:40.599152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:40.599181 (+ 29us) server_negotiation.cc:407] Connection header received
0620 07:10:40.599236 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:40.599242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:40.599470 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:40.599589 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:40.599992 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:40.601003 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:40.602167 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:40.602810 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:40.602906 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:40.603851 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:40.603859 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:40.603872 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:10:40.604452 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:40.604495 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:10:40.604513 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:40.604596 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:40.604718 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 07:10:40.604824 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":291,"threads_started":1}
I20260620 07:10:40.606366 16119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:40.687988 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 15997
I20260620 07:10:40.703125 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 15646
I20260620 07:10:40.704308 15707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:40.803774 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 15646
2026-06-20T07:10:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2406 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T07:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:10:40Z Disabled control of system clock
I20260620 07:10:40.872525 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:38267
--webserver_interface=127.6.53.126
--webserver_port=0
--builtin_ntp_servers=127.6.53.84:42683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:38267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 07:10:41.191627 16143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:41.191890 16143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:41.191973 16143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:41.199211 16143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:10:41.199324 16143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:41.199371 16143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:41.199436 16143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:10:41.199484 16143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:10:41.209306 16143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:42683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:38267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:38267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:41.210584 16143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:41.212248 16143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:41.224179 16148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:41.224200 16149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:41.224335 16151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:41.224877 16143 server_base.cc:1034] running on GCE node
I20260620 07:10:41.225776 16143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:41.227092 16143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:41.228329 16143 hybrid_clock.cc:648] HybridClock initialized: now 1781939441228290 us; error 60 us; skew 500 ppm
I20260620 07:10:41.228667 16143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:10:41.230974 16143 webserver.cc:460] Webserver started at http://127.6.53.126:37207/ using document root <none> and password file <none>
I20260620 07:10:41.231623 16143 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:41.231755 16143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:41.232074 16143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:41.234786 16143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "49693da8833945519b9b36a1f873fce9"
format_stamp: "Formatted at 2026-06-20 07:10:41 on dist-test-slave-4n6h"
server_key: "dd6cc4fafc7e4134df61fef5ee200792"
server_key_iv: "aa8ce720042fd90945e2f3bae01540d6"
server_key_version: "encryptionkey@0"
I20260620 07:10:41.235533 16143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "49693da8833945519b9b36a1f873fce9"
format_stamp: "Formatted at 2026-06-20 07:10:41 on dist-test-slave-4n6h"
server_key: "dd6cc4fafc7e4134df61fef5ee200792"
server_key_iv: "aa8ce720042fd90945e2f3bae01540d6"
server_key_version: "encryptionkey@0"
I20260620 07:10:41.240242 16143 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 07:10:41.243558 16157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:41.244760 16143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:10:41.244939 16143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "49693da8833945519b9b36a1f873fce9"
format_stamp: "Formatted at 2026-06-20 07:10:41 on dist-test-slave-4n6h"
server_key: "dd6cc4fafc7e4134df61fef5ee200792"
server_key_iv: "aa8ce720042fd90945e2f3bae01540d6"
server_key_version: "encryptionkey@0"
I20260620 07:10:41.245146 16143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:41.250525 16143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:41.251286 16143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:41.251587 16143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:41.268421 16143 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:38267
I20260620 07:10:41.268451 16208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:38267 every 8 connection(s)
I20260620 07:10:41.269703 16143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:10:41.271330 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 16143
I20260620 07:10:41.271561 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:10:41.272181 6357 external_mini_cluster.cc:1428] Setting key f746eed0d6546b1ef54bd4dfc40a2db8
I20260620 07:10:41.273636 16209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:41.281131 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:41.274081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40684 (local address 127.6.53.126:38267)
0620 07:10:41.275200 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:41.275214 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:41.275261 (+ 47us) server_negotiation.cc:407] Connection header received
0620 07:10:41.277112 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:41.277141 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:41.277674 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:41.278107 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:41.279130 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:41.279138 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:41.279156 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:10:41.279177 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:41.279276 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:10:41.279296 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:41.279539 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:41.279673 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 07:10:41.279932 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":408,"threads_started":1}
I20260620 07:10:41.287125 16209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9: Bootstrap starting.
I20260620 07:10:41.290194 16209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:41.291164 16209 log.cc:824] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:41.293642 16209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9: No bootstrap required, opened a new log
I20260620 07:10:41.300529 16209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49693da8833945519b9b36a1f873fce9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38267 } }
I20260620 07:10:41.300839 16209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:41.300908 16209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49693da8833945519b9b36a1f873fce9, State: Initialized, Role: FOLLOWER
I20260620 07:10:41.301425 16209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49693da8833945519b9b36a1f873fce9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38267 } }
I20260620 07:10:41.301651 16209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:10:41.301788 16209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:10:41.301996 16209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:41.304196 16209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49693da8833945519b9b36a1f873fce9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38267 } }
I20260620 07:10:41.304616 16209 leader_election.cc:302] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49693da8833945519b9b36a1f873fce9; no voters:
I20260620 07:10:41.305163 16209 leader_election.cc:288] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:10:41.305335 16214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:10:41.306107 16214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [term 1 LEADER]: Becoming Leader. State: Replica: 49693da8833945519b9b36a1f873fce9, State: Running, Role: LEADER
I20260620 07:10:41.306674 16214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49693da8833945519b9b36a1f873fce9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38267 } }
I20260620 07:10:41.307336 16209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:10:41.310050 16216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49693da8833945519b9b36a1f873fce9. Latest consensus state: current_term: 1 leader_uuid: "49693da8833945519b9b36a1f873fce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49693da8833945519b9b36a1f873fce9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38267 } } }
I20260620 07:10:41.310074 16215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49693da8833945519b9b36a1f873fce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49693da8833945519b9b36a1f873fce9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38267 } } }
I20260620 07:10:41.310540 16215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:41.310853 16216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:41.316728 16222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:10:41.322170 16222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:10:41.330489 16222 catalog_manager.cc:1347] Generated new cluster ID: 42f3c31bd9564368bcf7f19eeca8decb
I20260620 07:10:41.330610 16222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:10:41.385108 16222 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:10:41.385569 16222 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:10:41.393339 16222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49693da8833945519b9b36a1f873fce9: Generated new TSK 0
I20260620 07:10:41.394357 16222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:10:41.411854 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38267
--builtin_ntp_servers=127.6.53.84:42683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 07:10:41.760872 16233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:41.761128 16233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:41.761216 16233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:41.768157 16233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:41.768247 16233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:41.768350 16233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:10:41.776988 16233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:42683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:41.778137 16233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:41.779817 16233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:41.792548 16238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:41.792613 16241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:41.792531 16239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:41.798805 16233 server_base.cc:1034] running on GCE node
I20260620 07:10:41.799394 16233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:41.800289 16233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:41.801522 16233 hybrid_clock.cc:648] HybridClock initialized: now 1781939441801485 us; error 52 us; skew 500 ppm
I20260620 07:10:41.801849 16233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:10:41.804093 16233 webserver.cc:460] Webserver started at http://127.6.53.65:40145/ using document root <none> and password file <none>
I20260620 07:10:41.804797 16233 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:41.804919 16233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:41.805239 16233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:41.808037 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "74b40aacaa3549d789ece076189dec51"
format_stamp: "Formatted at 2026-06-20 07:10:41 on dist-test-slave-4n6h"
server_key: "39280b021fd5738cc84fa38bc23e7f01"
server_key_iv: "3ea39224860ceb00c02b027753da908a"
server_key_version: "encryptionkey@0"
I20260620 07:10:41.808908 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "74b40aacaa3549d789ece076189dec51"
format_stamp: "Formatted at 2026-06-20 07:10:41 on dist-test-slave-4n6h"
server_key: "39280b021fd5738cc84fa38bc23e7f01"
server_key_iv: "3ea39224860ceb00c02b027753da908a"
server_key_version: "encryptionkey@0"
I20260620 07:10:41.813670 16233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260620 07:10:41.817039 16247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:41.818358 16233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:10:41.818517 16233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "74b40aacaa3549d789ece076189dec51"
format_stamp: "Formatted at 2026-06-20 07:10:41 on dist-test-slave-4n6h"
server_key: "39280b021fd5738cc84fa38bc23e7f01"
server_key_iv: "3ea39224860ceb00c02b027753da908a"
server_key_version: "encryptionkey@0"
I20260620 07:10:41.818732 16233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:41.824064 16233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:41.824772 16233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:41.825021 16233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:41.825840 16233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:41.827334 16233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:41.827442 16233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:41.827576 16233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:41.827668 16233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:41.864513 16233 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:38419
I20260620 07:10:41.864564 16359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:38419 every 8 connection(s)
I20260620 07:10:41.865892 16233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:10:41.872102 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 16233
I20260620 07:10:41.872308 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:10:41.873039 6357 external_mini_cluster.cc:1428] Setting key 1302212835ff59a6e26589a1e814552b
I20260620 07:10:41.873943 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:41.868387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:32797 (local address 127.6.53.126:38267)
0620 07:10:41.868892 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:41.868899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:41.869929 (+ 1030us) server_negotiation.cc:407] Connection header received
0620 07:10:41.871632 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:41.871640 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:41.871710 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:41.871843 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:41.873190 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:41.873196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:41.873202 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:10:41.873224 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:41.873276 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:10:41.873292 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:41.873418 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:41.873586 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 07:10:41.873649 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":227,"threads_started":1}
I20260620 07:10:41.874893 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:41.868926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:38267 (local address 127.6.53.65:32797)
0620 07:10:41.869782 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:41.869822 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:41.871359 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:41.872015 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:41.872022 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:41.872107 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:41.872899 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:10:41.872902 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:41.872936 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:10:41.873016 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:41.873392 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:41.873399 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:41.873405 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 07:10:41.873602 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 07:10:41.873859 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":263,"threads_started":1}
I20260620 07:10:41.876073 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38267
--builtin_ntp_servers=127.6.53.84:42683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 07:10:41.876348 16360 heartbeater.cc:347] Connected to a master server at 127.6.53.126:38267
I20260620 07:10:41.876658 16360 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:41.877190 16360 heartbeater.cc:511] Master 127.6.53.126:38267 requested a full tablet report, sending...
I20260620 07:10:41.879225 16174 ts_manager.cc:194] Registered new tserver with Master: 74b40aacaa3549d789ece076189dec51 (127.6.53.65:38419)
W20260620 07:10:42.193912 16364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:42.194193 16364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:42.194356 16364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:42.201529 16364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:42.201625 16364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:42.201803 16364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:10:42.210951 16364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:42683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:42.212201 16364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:42.213891 16364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:42.226893 16369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:42.226893 16370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:42.226931 16372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:42.227893 16364 server_base.cc:1034] running on GCE node
I20260620 07:10:42.228430 16364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:42.229265 16364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:42.230475 16364 hybrid_clock.cc:648] HybridClock initialized: now 1781939442230436 us; error 53 us; skew 500 ppm
I20260620 07:10:42.230782 16364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:10:42.232897 16364 webserver.cc:460] Webserver started at http://127.6.53.66:37611/ using document root <none> and password file <none>
I20260620 07:10:42.233532 16364 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:42.233657 16364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:42.233975 16364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:42.236635 16364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "ab9ec16bcd454241ab562bb884e89db8"
format_stamp: "Formatted at 2026-06-20 07:10:42 on dist-test-slave-4n6h"
server_key: "e32e0342075aec431d7ed6bab98ecea9"
server_key_iv: "81c729893681a56f502dc5a74751a41e"
server_key_version: "encryptionkey@0"
I20260620 07:10:42.237388 16364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "ab9ec16bcd454241ab562bb884e89db8"
format_stamp: "Formatted at 2026-06-20 07:10:42 on dist-test-slave-4n6h"
server_key: "e32e0342075aec431d7ed6bab98ecea9"
server_key_iv: "81c729893681a56f502dc5a74751a41e"
server_key_version: "encryptionkey@0"
I20260620 07:10:42.242089 16364 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 07:10:42.245247 16378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:42.246400 16364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:10:42.246562 16364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "ab9ec16bcd454241ab562bb884e89db8"
format_stamp: "Formatted at 2026-06-20 07:10:42 on dist-test-slave-4n6h"
server_key: "e32e0342075aec431d7ed6bab98ecea9"
server_key_iv: "81c729893681a56f502dc5a74751a41e"
server_key_version: "encryptionkey@0"
I20260620 07:10:42.246766 16364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:42.252305 16364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:42.253021 16364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:42.253283 16364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:42.254128 16364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:42.255504 16364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:42.255623 16364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:42.255757 16364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:42.255848 16364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:42.290195 16364 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:44303
I20260620 07:10:42.290303 16490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:44303 every 8 connection(s)
I20260620 07:10:42.291450 16364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:10:42.297317 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 16364
I20260620 07:10:42.297510 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:10:42.298039 6357 external_mini_cluster.cc:1428] Setting key c90429682d70c6693754fc9093a4e483
I20260620 07:10:42.299160 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:42.293862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:60161 (local address 127.6.53.126:38267)
0620 07:10:42.294028 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:42.294033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:42.295349 (+ 1316us) server_negotiation.cc:407] Connection header received
0620 07:10:42.296869 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:42.296872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:42.296933 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:42.297018 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:42.298444 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:42.298448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:42.298451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:10:42.298467 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:42.298497 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:10:42.298509 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:42.298610 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:42.298858 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 07:10:42.298939 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:10:42.300185 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:42.294350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:38267 (local address 127.6.53.66:60161)
0620 07:10:42.295208 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:42.295246 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:42.296642 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:42.297272 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:42.297283 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:42.297402 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:42.298080 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:10:42.298086 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:42.298137 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:10:42.298241 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:42.298612 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:42.298623 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:42.298631 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:10:42.298886 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 07:10:42.299142 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":271,"threads_started":1}
I20260620 07:10:42.300956 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38267
--builtin_ntp_servers=127.6.53.84:42683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 07:10:42.301622 16491 heartbeater.cc:347] Connected to a master server at 127.6.53.126:38267
I20260620 07:10:42.301927 16491 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:42.302578 16491 heartbeater.cc:511] Master 127.6.53.126:38267 requested a full tablet report, sending...
I20260620 07:10:42.303913 16174 ts_manager.cc:194] Registered new tserver with Master: ab9ec16bcd454241ab562bb884e89db8 (127.6.53.66:44303)
W20260620 07:10:42.619700 16494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:42.619944 16494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:42.620029 16494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:42.627348 16494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:42.627441 16494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:42.627550 16494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:10:42.636499 16494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:42683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:42.637615 16494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:42.639256 16494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:42.652335 16499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:42.652371 16502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:42.652468 16500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:42.653029 16494 server_base.cc:1034] running on GCE node
I20260620 07:10:42.653620 16494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:42.654552 16494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:42.655771 16494 hybrid_clock.cc:648] HybridClock initialized: now 1781939442655737 us; error 52 us; skew 500 ppm
I20260620 07:10:42.656078 16494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:10:42.658360 16494 webserver.cc:460] Webserver started at http://127.6.53.67:34569/ using document root <none> and password file <none>
I20260620 07:10:42.659057 16494 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:42.659180 16494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:42.659514 16494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:42.662360 16494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "ffb3216087b949c48f252d8e0cb66c10"
format_stamp: "Formatted at 2026-06-20 07:10:42 on dist-test-slave-4n6h"
server_key: "e6f289fee202d09ff7b1b33781fd47cf"
server_key_iv: "67b5b2184c674a61ad0067bd2ba0564a"
server_key_version: "encryptionkey@0"
I20260620 07:10:42.663215 16494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "ffb3216087b949c48f252d8e0cb66c10"
format_stamp: "Formatted at 2026-06-20 07:10:42 on dist-test-slave-4n6h"
server_key: "e6f289fee202d09ff7b1b33781fd47cf"
server_key_iv: "67b5b2184c674a61ad0067bd2ba0564a"
server_key_version: "encryptionkey@0"
I20260620 07:10:42.668095 16494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 07:10:42.671309 16508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:42.672623 16494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:10:42.672791 16494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "ffb3216087b949c48f252d8e0cb66c10"
format_stamp: "Formatted at 2026-06-20 07:10:42 on dist-test-slave-4n6h"
server_key: "e6f289fee202d09ff7b1b33781fd47cf"
server_key_iv: "67b5b2184c674a61ad0067bd2ba0564a"
server_key_version: "encryptionkey@0"
I20260620 07:10:42.673010 16494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:42.678471 16494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:42.679196 16494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:42.679466 16494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:42.680300 16494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:42.681649 16494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:42.681770 16494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:42.681900 16494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:42.681988 16494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:42.714856 16494 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:40451
I20260620 07:10:42.714942 16620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:40451 every 8 connection(s)
I20260620 07:10:42.716133 16494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:10:42.723078 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 16494
I20260620 07:10:42.723268 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:10:42.723798 6357 external_mini_cluster.cc:1428] Setting key ccd8a3d4c828fab5dd9b991dabd76de5
I20260620 07:10:42.724452 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:42.718686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:53613 (local address 127.6.53.126:38267)
0620 07:10:42.718911 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:42.718918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:42.720226 (+ 1308us) server_negotiation.cc:407] Connection header received
0620 07:10:42.721954 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:42.721959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:42.722009 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:42.722098 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:42.723723 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:42.723728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:42.723731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:10:42.723745 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:42.723776 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:10:42.723787 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:42.723909 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:42.724116 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 07:10:42.724191 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:10:42.725449 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:42.719225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:38267 (local address 127.6.53.67:53613)
0620 07:10:42.720077 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:42.720115 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:42.721709 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:42.722339 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:42.722346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:42.722426 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:42.723396 (+ 970us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:10:42.723399 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:42.723450 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:10:42.723552 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:42.723890 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:42.723903 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:42.723911 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:10:42.724216 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 07:10:42.724467 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":263,"threads_started":1}
I20260620 07:10:42.726743 16621 heartbeater.cc:347] Connected to a master server at 127.6.53.126:38267
I20260620 07:10:42.727025 16621 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:42.727569 16621 heartbeater.cc:511] Master 127.6.53.126:38267 requested a full tablet report, sending...
I20260620 07:10:42.729039 16174 ts_manager.cc:194] Registered new tserver with Master: ffb3216087b949c48f252d8e0cb66c10 (127.6.53.67:40451)
I20260620 07:10:42.739173 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 07:10:42.744936 16362 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:10:42.743433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.6.53.126:38267)
0620 07:10:42.743608 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:42.743615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:42.744024 (+ 409us) server_negotiation.cc:407] Connection header received
0620 07:10:42.744209 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:42.744215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:42.744284 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:42.744388 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:42.744695 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47130: BlockingRecv error: recv got EOF from 127.0.0.1:47130 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:10:42.745795 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 16233
I20260620 07:10:42.749033 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:42.746779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.6.53.65:38419)
0620 07:10:42.747290 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:42.747296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:42.747319 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:10:42.747369 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:42.747375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:42.747543 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:42.747657 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:42.748484 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:42.748490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:42.748500 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:10:42.748517 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:42.748553 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:10:42.748566 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:42.748663 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:42.748740 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 07:10:42.748812 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":260,"threads_started":1}
I20260620 07:10:42.750185 16355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:42.830168 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 16233
I20260620 07:10:42.847117 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 16364
I20260620 07:10:42.850929 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:42.848129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.6.53.66:44303)
0620 07:10:42.848604 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:42.848624 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:42.848649 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:10:42.848725 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:42.848730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:42.848994 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:42.849158 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:42.850136 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:42.850142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:42.850152 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:10:42.850164 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:42.850203 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:10:42.850216 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:42.850398 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:42.850507 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 07:10:42.850605 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":237,"threads_started":1}
I20260620 07:10:42.852051 16486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:42.932216 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 16364
I20260620 07:10:42.948844 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 16494
I20260620 07:10:42.953312 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:42.950143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45296 (local address 127.6.53.67:40451)
0620 07:10:42.950870 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:42.950877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:42.950908 (+ 31us) server_negotiation.cc:407] Connection header received
0620 07:10:42.950966 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:42.950972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:42.951177 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:42.951351 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:42.952464 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:42.952471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:42.952481 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:10:42.952521 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:42.952571 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:10:42.952585 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:42.952833 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:42.952955 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 07:10:42.953056 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":410,"threads_started":1}
I20260620 07:10:42.954614 16616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:43.038806 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 16494
I20260620 07:10:43.054776 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 16143
I20260620 07:10:43.055848 16204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:43.155603 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 16143
2026-06-20T07:10:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2351 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:10:43 dist-test-slave-4n6h krb5kdc[16639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:10:43 dist-test-slave-4n6h krb5kdc[16639](info): set up 2 sockets
Jun 20 07:10:43 dist-test-slave-4n6h krb5kdc[16639](info): commencing operation
krb5kdc: starting...
W20260620 07:10:44.267249 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:10:44 dist-test-slave-4n6h krb5kdc[16639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:10:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:10:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:10:44.489984 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:42233
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:39523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:42233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:44.825394 16652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:44.825685 16652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:44.825923 16652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:44.833156 16652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:10:44.833289 16652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:44.833405 16652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:44.833539 16652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:10:44.833633 16652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:10:44.843585 16652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:39523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:42233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:42233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:44.844858 16652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:44.846201 16652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:44.858318 16657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:44.858281 16660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:44.858356 16658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:44.859117 16652 server_base.cc:1034] running on GCE node
I20260620 07:10:44.859826 16652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:44.861155 16652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:44.862360 16652 hybrid_clock.cc:648] HybridClock initialized: now 1781939444862312 us; error 71 us; skew 500 ppm
I20260620 07:10:44.862695 16652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:44 dist-test-slave-4n6h krb5kdc[16639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939444, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:44.865538 16652 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:10:44.866981 16652 webserver.cc:460] Webserver started at http://127.6.53.126:45647/ using document root <none> and password file <none>
I20260620 07:10:44.867609 16652 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:44.867758 16652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:44.868050 16652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:44.870929 16652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "6ed9445c768c48bebeb21689a1c12d8d"
format_stamp: "Formatted at 2026-06-20 07:10:44 on dist-test-slave-4n6h"
server_key: "ca678201c87f71fb16c044856262695c"
server_key_iv: "8a74bfbc09c1a269b718bc93a4415491"
server_key_version: "encryptionkey@0"
I20260620 07:10:44.871668 16652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "6ed9445c768c48bebeb21689a1c12d8d"
format_stamp: "Formatted at 2026-06-20 07:10:44 on dist-test-slave-4n6h"
server_key: "ca678201c87f71fb16c044856262695c"
server_key_iv: "8a74bfbc09c1a269b718bc93a4415491"
server_key_version: "encryptionkey@0"
I20260620 07:10:44.876555 16652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:10:44.880082 16667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:44.881399 16652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:10:44.881572 16652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "6ed9445c768c48bebeb21689a1c12d8d"
format_stamp: "Formatted at 2026-06-20 07:10:44 on dist-test-slave-4n6h"
server_key: "ca678201c87f71fb16c044856262695c"
server_key_iv: "8a74bfbc09c1a269b718bc93a4415491"
server_key_version: "encryptionkey@0"
I20260620 07:10:44.881774 16652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:44.903765 16652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:44.907703 16652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:44.907982 16652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:44.928555 16652 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:42233
I20260620 07:10:44.928582 16718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:42233 every 8 connection(s)
I20260620 07:10:44.930027 16652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:10:44.930858 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 16652
I20260620 07:10:44.931118 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:10:44.931843 6357 external_mini_cluster.cc:1428] Setting key e04da82be2555bd13cea6eaf48484376
I20260620 07:10:44.934510 16719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:44.945540 16719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d: Bootstrap starting.
I20260620 07:10:44.948626 16719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:44.949687 16719 log.cc:824] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:10:44 dist-test-slave-4n6h krb5kdc[16639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:10:44.953327 16719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d: No bootstrap required, opened a new log
I20260620 07:10:44.960299 16722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:44.933936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.6.53.126:42233)
0620 07:10:44.934792 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:44.934804 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:44.934842 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:10:44.936404 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:44.936422 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:44.936796 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:44.937231 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:44.938772 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:44.940070 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:44.941325 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:44.941977 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:44.942174 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:44.952971 (+ 10797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:44.952979 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:44.952995 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:44.953034 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:44.956202 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:44.957104 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:44.957110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:44.957116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:44.957270 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:44.957694 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:44.957697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:44.957699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:44.958181 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:44.958494 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:44.958913 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 07:10:44.959264 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":645,"thread_start_us":395,"threads_started":1}
I20260620 07:10:44.963608 16719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed9445c768c48bebeb21689a1c12d8d" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42233 } }
I20260620 07:10:44.964017 16719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:44.964186 16719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed9445c768c48bebeb21689a1c12d8d, State: Initialized, Role: FOLLOWER
I20260620 07:10:44.964792 16719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed9445c768c48bebeb21689a1c12d8d" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42233 } }
I20260620 07:10:44.965060 16719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:10:44.965206 16719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:10:44.965385 16719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:44.967516 16719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed9445c768c48bebeb21689a1c12d8d" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42233 } }
I20260620 07:10:44.967962 16719 leader_election.cc:302] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ed9445c768c48bebeb21689a1c12d8d; no voters:
I20260620 07:10:44.968523 16719 leader_election.cc:288] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:10:44.968658 16724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:10:44.970062 16724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [term 1 LEADER]: Becoming Leader. State: Replica: 6ed9445c768c48bebeb21689a1c12d8d, State: Running, Role: LEADER
I20260620 07:10:44.970528 16719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:10:44.970615 16724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed9445c768c48bebeb21689a1c12d8d" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42233 } }
I20260620 07:10:44.973495 16726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ed9445c768c48bebeb21689a1c12d8d. Latest consensus state: current_term: 1 leader_uuid: "6ed9445c768c48bebeb21689a1c12d8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed9445c768c48bebeb21689a1c12d8d" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42233 } } }
I20260620 07:10:44.973834 16726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:44.973601 16725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ed9445c768c48bebeb21689a1c12d8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed9445c768c48bebeb21689a1c12d8d" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 42233 } } }
I20260620 07:10:44.974032 16725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d [sys.catalog]: This master's current role is: LEADER
I20260620 07:10:44.983232 16732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:10:44.987478 16732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:10:44.995205 16732 catalog_manager.cc:1347] Generated new cluster ID: 94002b25d82c483cb4ecf99cb2eb9d6d
I20260620 07:10:44.995328 16732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:10:45.008246 16732 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:10:45.009503 16732 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:10:45.018806 16732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d: Generated new TSK 0
I20260620 07:10:45.019579 16732 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:10:45.073933 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42233
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:39523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:45.393227 16745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:45.393482 16745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:45.393577 16745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:45.400738 16745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:45.400838 16745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:45.400954 16745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:10:45.410138 16745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:39523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:45.411345 16745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:45.412545 16745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:45.425045 16750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:45.425087 16751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:45.425109 16753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:45.425913 16745 server_base.cc:1034] running on GCE node
I20260620 07:10:45.426496 16745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:45.427410 16745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:45.428619 16745 hybrid_clock.cc:648] HybridClock initialized: now 1781939445428566 us; error 30 us; skew 500 ppm
I20260620 07:10:45.428879 16745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:45 dist-test-slave-4n6h krb5kdc[16639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939445, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:45.431859 16745 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:10:45.433176 16745 webserver.cc:460] Webserver started at http://127.6.53.65:36145/ using document root <none> and password file <none>
I20260620 07:10:45.433766 16745 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:45.433862 16745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:45.434185 16745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:45.437007 16745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "375e2d8af3124e5cbd08b2fd878e8300"
format_stamp: "Formatted at 2026-06-20 07:10:45 on dist-test-slave-4n6h"
server_key: "013a9b0656141480d468a8405b27e962"
server_key_iv: "aa6a5061b1a70b0611ffa5e77fe527c7"
server_key_version: "encryptionkey@0"
I20260620 07:10:45.437680 16745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "375e2d8af3124e5cbd08b2fd878e8300"
format_stamp: "Formatted at 2026-06-20 07:10:45 on dist-test-slave-4n6h"
server_key: "013a9b0656141480d468a8405b27e962"
server_key_iv: "aa6a5061b1a70b0611ffa5e77fe527c7"
server_key_version: "encryptionkey@0"
I20260620 07:10:45.442395 16745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260620 07:10:45.445643 16760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:45.446975 16745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:10:45.447151 16745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "375e2d8af3124e5cbd08b2fd878e8300"
format_stamp: "Formatted at 2026-06-20 07:10:45 on dist-test-slave-4n6h"
server_key: "013a9b0656141480d468a8405b27e962"
server_key_iv: "aa6a5061b1a70b0611ffa5e77fe527c7"
server_key_version: "encryptionkey@0"
I20260620 07:10:45.447335 16745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:45.480934 16745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:45.485263 16745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:45.485558 16745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:45.486622 16745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:45.487892 16745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:45.487962 16745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:45.488123 16745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:45.488186 16745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:45.520716 16745 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:33391
I20260620 07:10:45.520839 16872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:33391 every 8 connection(s)
I20260620 07:10:45.521943 16745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:10:45.524703 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 16745
I20260620 07:10:45.524928 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:10:45.525831 6357 external_mini_cluster.cc:1428] Setting key 2b10b12c7c3e3eaafe42826a710dc348
Jun 20 07:10:45 dist-test-slave-4n6h krb5kdc[16639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939445, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:10:45.540975 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:45.524340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:58255 (local address 127.6.53.126:42233)
0620 07:10:45.524767 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:45.524791 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:45.525854 (+ 1063us) server_negotiation.cc:407] Connection header received
0620 07:10:45.527749 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:45.527754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:45.527835 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:45.527943 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:45.529367 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:45.530050 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:45.531615 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:45.531995 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:45.532134 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:45.535573 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:45.535578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:45.535581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:45.535617 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:45.538087 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:45.538791 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:45.538799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:45.538807 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:45.538914 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:45.539352 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:45.539356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:45.539359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:45.539595 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:45.539751 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:45.540379 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 07:10:45.540605 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":228,"threads_started":1}
I20260620 07:10:45.541882 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:45.524972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42233 (local address 127.6.53.65:58255)
0620 07:10:45.525709 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:45.525744 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:45.527506 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:45.528162 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:45.528169 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:45.528647 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:45.529210 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:45.529226 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:45.530212 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:45.530216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:45.531452 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:45.531464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:45.532228 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:45.532235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:45.532459 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:45.533170 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:45.533190 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:45.535364 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:45.538227 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:45.538235 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:45.538270 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:45.538665 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:45.539029 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:45.539033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:45.539037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:45.539224 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:45.539724 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:45.539732 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:45.540029 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 07:10:45.540279 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 07:10:45.540794 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":304,"threads_started":1}
I20260620 07:10:45.543382 16873 heartbeater.cc:347] Connected to a master server at 127.6.53.126:42233
I20260620 07:10:45.543730 16873 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:45.544669 16873 heartbeater.cc:511] Master 127.6.53.126:42233 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
I20260620 07:10:45.546904 16684 ts_manager.cc:194] Registered new tserver with Master: 375e2d8af3124e5cbd08b2fd878e8300 (127.6.53.65:33391)
I20260620 07:10:45.548369 16684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:58255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:10:45.575196 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42233
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:39523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:45.898608 16879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:45.898959 16879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:45.899140 16879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:45.906206 16879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:45.906369 16879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:45.906567 16879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:10:45.916769 16879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:39523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:45.918044 16879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:45.919337 16879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:45.932359 16885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:45.932546 16887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:45.932971 16884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:45.933712 16879 server_base.cc:1034] running on GCE node
I20260620 07:10:45.934450 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:45.935564 16879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:45.936797 16879 hybrid_clock.cc:648] HybridClock initialized: now 1781939445936691 us; error 97 us; skew 500 ppm
I20260620 07:10:45.937150 16879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:45 dist-test-slave-4n6h krb5kdc[16639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939445, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:45.940841 16879 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:10:45.942436 16879 webserver.cc:460] Webserver started at http://127.6.53.66:35637/ using document root <none> and password file <none>
I20260620 07:10:45.943113 16879 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:45.943239 16879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:45.943542 16879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:45.946481 16879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "8474cedcfdc54d9bb3f78dd9e49d5048"
format_stamp: "Formatted at 2026-06-20 07:10:45 on dist-test-slave-4n6h"
server_key: "7cc39ea64532641093c67fc56507b1e5"
server_key_iv: "16d06d353ae8f3472e383194c1b5882c"
server_key_version: "encryptionkey@0"
I20260620 07:10:45.947247 16879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "8474cedcfdc54d9bb3f78dd9e49d5048"
format_stamp: "Formatted at 2026-06-20 07:10:45 on dist-test-slave-4n6h"
server_key: "7cc39ea64532641093c67fc56507b1e5"
server_key_iv: "16d06d353ae8f3472e383194c1b5882c"
server_key_version: "encryptionkey@0"
I20260620 07:10:45.952446 16879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:10:45.955868 16894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:45.957185 16879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:10:45.957368 16879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "8474cedcfdc54d9bb3f78dd9e49d5048"
format_stamp: "Formatted at 2026-06-20 07:10:45 on dist-test-slave-4n6h"
server_key: "7cc39ea64532641093c67fc56507b1e5"
server_key_iv: "16d06d353ae8f3472e383194c1b5882c"
server_key_version: "encryptionkey@0"
I20260620 07:10:45.957568 16879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:45.979678 16879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:45.983706 16879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:45.983990 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:45.984965 16879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:45.986346 16879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:45.986416 16879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:45.986568 16879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:45.986618 16879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:46.019502 16879 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:41531
I20260620 07:10:46.019589 17006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:41531 every 8 connection(s)
I20260620 07:10:46.020751 16879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:10:46.026274 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 16879
I20260620 07:10:46.026475 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:10:46.027038 6357 external_mini_cluster.cc:1428] Setting key 56e9b48c6f184e3ab9ec55ef4f2d9bcf
Jun 20 07:10:46 dist-test-slave-4n6h krb5kdc[16639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939445, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:10:46.041616 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.023255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:52621 (local address 127.6.53.126:42233)
0620 07:10:46.023455 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:46.023461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:46.024620 (+ 1159us) server_negotiation.cc:407] Connection header received
0620 07:10:46.026620 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:46.026626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:46.026703 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:46.026789 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:46.028632 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.029338 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.030967 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.031385 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.031521 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.035086 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:46.035090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:46.035094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:46.035122 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:46.038385 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:46.039073 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:46.039081 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:46.039088 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:46.039235 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:46.039624 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:46.039631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:46.039644 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:46.040001 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:46.040149 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:46.040976 (+ 827us) server_negotiation.cc:299] Negotiation successful
0620 07:10:46.041309 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:10:46.042500 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.023780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42233 (local address 127.6.53.66:52621)
0620 07:10:46.024457 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:46.024498 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:46.026178 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:46.027053 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:46.027065 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:46.027747 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:46.028470 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:46.028487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.029493 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.029498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:46.030820 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:46.030832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.031605 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.031612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:46.031838 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.032522 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:46.032543 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:46.034864 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:46.038511 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:46.038518 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:46.038530 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:46.038927 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:46.039340 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:46.039343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:46.039346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:46.039509 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:46.040115 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:46.040121 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:46.040453 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 07:10:46.040794 (+ 341us) client_negotiation.cc:240] Negotiation successful
0620 07:10:46.041289 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":269,"threads_started":1}
I20260620 07:10:46.043879 17007 heartbeater.cc:347] Connected to a master server at 127.6.53.126:42233
I20260620 07:10:46.044201 17007 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:46.045033 17007 heartbeater.cc:511] Master 127.6.53.126:42233 requested a full tablet report, sending...
I20260620 07:10:46.046571 16684 ts_manager.cc:194] Registered new tserver with Master: 8474cedcfdc54d9bb3f78dd9e49d5048 (127.6.53.66:41531)
I20260620 07:10:46.047433 16684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:52621
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:10:46.077333 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42233
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:39523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:10:46.404533 17012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:10:46.404878 17012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:10:46.405081 17012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:10:46.412539 17012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:10:46.412709 17012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:10:46.412900 17012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:10:46.422761 17012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:39523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:42233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:10:46.424059 17012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:10:46.425351 17012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:10:46.440927 17018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:46.440888 17017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:10:46.441345 17020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:10:46.442020 17012 server_base.cc:1034] running on GCE node
I20260620 07:10:46.442649 17012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:10:46.443555 17012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:10:46.444787 17012 hybrid_clock.cc:648] HybridClock initialized: now 1781939446444749 us; error 53 us; skew 500 ppm
I20260620 07:10:46.445108 17012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:10:46 dist-test-slave-4n6h krb5kdc[16639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939446, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:10:46.448385 17012 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:10:46.449896 17012 webserver.cc:460] Webserver started at http://127.6.53.67:36601/ using document root <none> and password file <none>
I20260620 07:10:46.450651 17012 fs_manager.cc:359] Metadata directory not provided
I20260620 07:10:46.450795 17012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:10:46.451114 17012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:10:46.453927 17012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "39fe68012d2f47b3b54973f61505c4c0"
format_stamp: "Formatted at 2026-06-20 07:10:46 on dist-test-slave-4n6h"
server_key: "2b7a19937b8a54d5af2c67ddeb37f485"
server_key_iv: "b4c5609849c0df46cecc4522e254b0da"
server_key_version: "encryptionkey@0"
I20260620 07:10:46.454746 17012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "39fe68012d2f47b3b54973f61505c4c0"
format_stamp: "Formatted at 2026-06-20 07:10:46 on dist-test-slave-4n6h"
server_key: "2b7a19937b8a54d5af2c67ddeb37f485"
server_key_iv: "b4c5609849c0df46cecc4522e254b0da"
server_key_version: "encryptionkey@0"
I20260620 07:10:46.459384 17012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:10:46.462764 17027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:46.463979 17012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:10:46.464108 17012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "39fe68012d2f47b3b54973f61505c4c0"
format_stamp: "Formatted at 2026-06-20 07:10:46 on dist-test-slave-4n6h"
server_key: "2b7a19937b8a54d5af2c67ddeb37f485"
server_key_iv: "b4c5609849c0df46cecc4522e254b0da"
server_key_version: "encryptionkey@0"
I20260620 07:10:46.464252 17012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:10:46.478423 17012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:10:46.482741 17012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:10:46.482966 17012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:10:46.483811 17012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:10:46.485059 17012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:10:46.485137 17012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:46.485213 17012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:10:46.485258 17012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:10:46.516831 17012 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:35549
I20260620 07:10:46.516875 17139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:35549 every 8 connection(s)
I20260620 07:10:46.518047 17012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:10:46.528072 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 17012
I20260620 07:10:46.528276 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:10:46.528698 6357 external_mini_cluster.cc:1428] Setting key 015033b951a07eff85064df7c11ddeaf
Jun 20 07:10:46 dist-test-slave-4n6h krb5kdc[16639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939446, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:10:46.537747 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.520627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:34121 (local address 127.6.53.126:42233)
0620 07:10:46.520828 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:46.520832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:46.522036 (+ 1204us) server_negotiation.cc:407] Connection header received
0620 07:10:46.523864 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:46.523869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:46.523961 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:46.524069 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:46.525439 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.526174 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.527712 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.528137 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.528406 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.532384 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:46.532389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:46.532406 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:46.532436 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:46.535073 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:46.535717 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:46.535721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:46.535723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:46.535883 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:46.536337 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:46.536340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:46.536343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:46.536607 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:46.536717 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:46.537345 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 07:10:46.537496 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:10:46.538672 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.521135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:42233 (local address 127.6.53.67:34121)
0620 07:10:46.521884 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:46.521918 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:46.523606 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:46.524289 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:46.524297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:46.524768 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:10:46.525281 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:46.525296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.526344 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.526350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:46.527534 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:46.527546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.528478 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.528488 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:46.528738 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.529548 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:10:46.529569 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:10:46.532207 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:10:46.535200 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:46.535207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:46.535217 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:46.535584 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:46.535986 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:10:46.535989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:10:46.535991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:10:46.536197 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:10:46.536711 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:10:46.536716 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:10:46.536988 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 07:10:46.537252 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 07:10:46.537591 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":301,"threads_started":1}
I20260620 07:10:46.539987 17140 heartbeater.cc:347] Connected to a master server at 127.6.53.126:42233
I20260620 07:10:46.540288 17140 heartbeater.cc:464] Registering TS with master...
I20260620 07:10:46.540977 17140 heartbeater.cc:511] Master 127.6.53.126:42233 requested a full tablet report, sending...
I20260620 07:10:46.542374 16684 ts_manager.cc:194] Registered new tserver with Master: 39fe68012d2f47b3b54973f61505c4c0 (127.6.53.67:35549)
I20260620 07:10:46.543497 16684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:34121
I20260620 07:10:46.544229 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:10:46.551318 16873 heartbeater.cc:503] Master 127.6.53.126:42233 was elected leader, sending a full tablet report...
I20260620 07:10:46.561184 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.548695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39234 (local address 127.6.53.126:42233)
0620 07:10:46.548845 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:46.548849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:46.549363 (+ 514us) server_negotiation.cc:407] Connection header received
0620 07:10:46.549553 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:46.549558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:46.549639 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:46.549744 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:46.551284 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.552295 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.553623 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.554017 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.554116 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.556200 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:46.556204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:46.556207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:46.556235 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:46.558803 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:46.559382 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:46.559385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:46.559387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:46.559501 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:46.559797 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:46.559800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:46.559802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:46.560125 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:46.560318 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:46.560630 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 07:10:46.560870 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:10:46.565052 16683 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:10:46.572367 16683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:10:46.595789 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.584366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.6.53.65:33391)
0620 07:10:46.585611 (+ 1245us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:46.585620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:46.586133 (+ 513us) server_negotiation.cc:407] Connection header received
0620 07:10:46.586421 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:46.586428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:46.586625 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:46.586763 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:46.587704 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.589315 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.592701 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.594139 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.594242 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.594490 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:46.595226 (+ 736us) server_negotiation.cc:299] Negotiation successful
0620 07:10:46.595450 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1086,"thread_start_us":965,"threads_started":1}
I20260620 07:10:46.596401 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.583858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:33391 (local address 127.0.0.1:45474)
0620 07:10:46.586016 (+ 2158us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:46.586034 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:46.586241 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:46.587029 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:46.587035 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:46.587065 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:46.587486 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:46.587501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.589476 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.589479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:46.592521 (+ 3042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:46.592536 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.594699 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.594704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:46.594899 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.595033 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 07:10:46.595162 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 07:10:46.595535 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2045,"mutex_wait_us":235,"thread_start_us":621,"threads_started":1}
I20260620 07:10:46.596642 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.583734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:35549 (local address 127.0.0.1:44264)
0620 07:10:46.584513 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:46.584560 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:46.584712 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:46.586705 (+ 1993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:46.586714 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:46.586750 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:46.587196 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:46.587209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.589470 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.589475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:46.592521 (+ 3046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:46.592536 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.594671 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.594683 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:46.594905 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.595033 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 07:10:46.595162 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 07:10:46.595977 (+ 815us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":531,"thread_start_us":2033,"threads_started":1}
I20260620 07:10:46.596788 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.583984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44264 (local address 127.6.53.67:35549)
0620 07:10:46.585978 (+ 1994us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:46.585985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:46.586018 (+ 33us) server_negotiation.cc:407] Connection header received
0620 07:10:46.586202 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:46.586210 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:46.586473 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:46.587402 (+ 929us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:46.587682 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.589317 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.592701 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.594138 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.594276 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.594489 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:46.596229 (+ 1740us) server_negotiation.cc:299] Negotiation successful
0620 07:10:46.596396 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1838,"thread_start_us":1691,"threads_started":1}
I20260620 07:10:46.600365 17075 tablet_service.cc:1467] Processing CreateTablet for tablet fc9e9254dda449efbeb91d06fc720454 (DEFAULT_TABLE table=test-table [id=fe5f724eac744f9fb5ddc830c0000018]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:46.600365 16808 tablet_service.cc:1467] Processing CreateTablet for tablet fc9e9254dda449efbeb91d06fc720454 (DEFAULT_TABLE table=test-table [id=fe5f724eac744f9fb5ddc830c0000018]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:46.601770 16808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc9e9254dda449efbeb91d06fc720454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:46.601785 17075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc9e9254dda449efbeb91d06fc720454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:46.603674 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.586183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:41531 (local address 127.0.0.1:48392)
0620 07:10:46.596475 (+ 10292us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:46.597537 (+ 1062us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:46.597676 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:46.598195 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:46.598199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:46.598222 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:46.598563 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:46.598572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.599815 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.599822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:46.601265 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:46.601276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.602919 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.602926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:46.603089 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.603122 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:10:46.603226 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 07:10:46.603346 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10138,"thread_start_us":8751,"threads_started":1}
I20260620 07:10:46.603722 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.586470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.6.53.66:41531)
0620 07:10:46.595615 (+ 9145us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:46.595625 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:46.597633 (+ 2008us) server_negotiation.cc:407] Connection header received
0620 07:10:46.597846 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:46.597851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:46.598011 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:46.598208 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:46.598727 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.599681 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.601402 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.602716 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.602859 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.603046 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:46.603264 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 07:10:46.603413 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8981,"thread_start_us":8844,"threads_started":1}
I20260620 07:10:46.606685 16942 tablet_service.cc:1467] Processing CreateTablet for tablet fc9e9254dda449efbeb91d06fc720454 (DEFAULT_TABLE table=test-table [id=fe5f724eac744f9fb5ddc830c0000018]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:10:46.607945 16942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc9e9254dda449efbeb91d06fc720454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:10:46.616420 17158 tablet_bootstrap.cc:492] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300: Bootstrap starting.
I20260620 07:10:46.617216 17159 tablet_bootstrap.cc:492] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0: Bootstrap starting.
I20260620 07:10:46.619050 17160 tablet_bootstrap.cc:492] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048: Bootstrap starting.
I20260620 07:10:46.620623 17158 tablet_bootstrap.cc:654] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:46.621702 17159 tablet_bootstrap.cc:654] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:46.621774 17158 log.cc:824] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:46.622435 17160 tablet_bootstrap.cc:654] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048: Neither blocks nor log segments found. Creating new log.
I20260620 07:10:46.622906 17159 log.cc:824] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:46.623584 17160 log.cc:824] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048: Log is configured to *not* fsync() on all Append() calls
I20260620 07:10:46.625049 17158 tablet_bootstrap.cc:492] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300: No bootstrap required, opened a new log
I20260620 07:10:46.625324 17158 ts_tablet_manager.cc:1397] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 07:10:46.626119 17160 tablet_bootstrap.cc:492] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048: No bootstrap required, opened a new log
I20260620 07:10:46.626382 17160 ts_tablet_manager.cc:1397] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260620 07:10:46.626600 17159 tablet_bootstrap.cc:492] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0: No bootstrap required, opened a new log
I20260620 07:10:46.626829 17159 ts_tablet_manager.cc:1397] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 07:10:46.633059 17160 raft_consensus.cc:348] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fe68012d2f47b3b54973f61505c4c0" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35549 } } peers { permanent_uuid: "375e2d8af3124e5cbd08b2fd878e8300" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33391 } } peers { permanent_uuid: "8474cedcfdc54d9bb3f78dd9e49d5048" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 41531 } }
I20260620 07:10:46.633483 17160 raft_consensus.cc:374] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:46.633579 17160 raft_consensus.cc:729] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8474cedcfdc54d9bb3f78dd9e49d5048, State: Initialized, Role: FOLLOWER
I20260620 07:10:46.634285 17160 consensus_queue.cc:260] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fe68012d2f47b3b54973f61505c4c0" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35549 } } peers { permanent_uuid: "375e2d8af3124e5cbd08b2fd878e8300" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33391 } } peers { permanent_uuid: "8474cedcfdc54d9bb3f78dd9e49d5048" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 41531 } }
I20260620 07:10:46.635394 17007 heartbeater.cc:503] Master 127.6.53.126:42233 was elected leader, sending a full tablet report...
I20260620 07:10:46.636027 17160 ts_tablet_manager.cc:1428] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20260620 07:10:46.636539 17158 raft_consensus.cc:348] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fe68012d2f47b3b54973f61505c4c0" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35549 } } peers { permanent_uuid: "375e2d8af3124e5cbd08b2fd878e8300" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33391 } } peers { permanent_uuid: "8474cedcfdc54d9bb3f78dd9e49d5048" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 41531 } }
I20260620 07:10:46.636986 17158 raft_consensus.cc:374] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:46.637085 17158 raft_consensus.cc:729] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375e2d8af3124e5cbd08b2fd878e8300, State: Initialized, Role: FOLLOWER
I20260620 07:10:46.637151 17159 raft_consensus.cc:348] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fe68012d2f47b3b54973f61505c4c0" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35549 } } peers { permanent_uuid: "375e2d8af3124e5cbd08b2fd878e8300" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33391 } } peers { permanent_uuid: "8474cedcfdc54d9bb3f78dd9e49d5048" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 41531 } }
I20260620 07:10:46.637507 17159 raft_consensus.cc:374] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:10:46.637605 17159 raft_consensus.cc:729] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39fe68012d2f47b3b54973f61505c4c0, State: Initialized, Role: FOLLOWER
I20260620 07:10:46.637674 17158 consensus_queue.cc:260] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fe68012d2f47b3b54973f61505c4c0" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35549 } } peers { permanent_uuid: "375e2d8af3124e5cbd08b2fd878e8300" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33391 } } peers { permanent_uuid: "8474cedcfdc54d9bb3f78dd9e49d5048" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 41531 } }
I20260620 07:10:46.638149 17159 consensus_queue.cc:260] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fe68012d2f47b3b54973f61505c4c0" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35549 } } peers { permanent_uuid: "375e2d8af3124e5cbd08b2fd878e8300" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33391 } } peers { permanent_uuid: "8474cedcfdc54d9bb3f78dd9e49d5048" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 41531 } }
I20260620 07:10:46.643648 17158 ts_tablet_manager.cc:1428] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260620 07:10:46.643834 17159 ts_tablet_manager.cc:1428] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260620 07:10:46.643934 17140 heartbeater.cc:503] Master 127.6.53.126:42233 was elected leader, sending a full tablet report...
I20260620 07:10:46.685593 17164 raft_consensus.cc:482] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:10:46.685878 17164 raft_consensus.cc:504] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fe68012d2f47b3b54973f61505c4c0" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35549 } } peers { permanent_uuid: "375e2d8af3124e5cbd08b2fd878e8300" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33391 } } peers { permanent_uuid: "8474cedcfdc54d9bb3f78dd9e49d5048" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 41531 } }
I20260620 07:10:46.687620 17164 leader_election.cc:288] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39fe68012d2f47b3b54973f61505c4c0 (127.6.53.67:35549), 375e2d8af3124e5cbd08b2fd878e8300 (127.6.53.65:33391)
I20260620 07:10:46.694569 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.687999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:53487 (local address 127.6.53.67:35549)
0620 07:10:46.688224 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:46.688231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:46.689231 (+ 1000us) server_negotiation.cc:407] Connection header received
0620 07:10:46.689335 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:46.689340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:46.689417 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:46.689503 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:46.690205 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.691032 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.692816 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.693544 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.693640 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.693787 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:46.694165 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 07:10:46.694307 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 07:10:46.694581 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.687979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:41951 (local address 127.6.53.65:33391)
0620 07:10:46.688176 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:46.688182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:46.688804 (+ 622us) server_negotiation.cc:407] Connection header received
0620 07:10:46.689105 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:46.689110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:46.689169 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:46.689254 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:10:46.690086 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.690974 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.692776 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.693555 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.693721 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.693829 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:46.694189 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 07:10:46.694339 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:10:46.694705 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.688398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:35549 (local address 127.6.53.66:53487)
0620 07:10:46.689123 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:46.689138 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:46.689241 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:46.689538 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:46.689540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:46.689560 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:46.690052 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:46.690065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.691212 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.691216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:46.692637 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:46.692649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.693787 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.693797 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:46.693998 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.694028 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:10:46.694100 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 07:10:46.694250 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":51,"thread_start_us":415,"threads_started":1}
I20260620 07:10:46.694923 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.687879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:33391 (local address 127.6.53.66:41951)
0620 07:10:46.688663 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:46.688678 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:10:46.688868 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:10:46.689457 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:10:46.689463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:10:46.689493 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:10:46.689925 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:46.689934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.691111 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.691118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:46.692616 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:10:46.692628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.693815 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.693822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:10:46.694010 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.694037 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 07:10:46.694141 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 07:10:46.694282 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":43,"thread_start_us":487,"threads_started":1}
I20260620 07:10:46.695752 17095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9e9254dda449efbeb91d06fc720454" candidate_uuid: "8474cedcfdc54d9bb3f78dd9e49d5048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39fe68012d2f47b3b54973f61505c4c0" is_pre_election: true
I20260620 07:10:46.695818 16828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9e9254dda449efbeb91d06fc720454" candidate_uuid: "8474cedcfdc54d9bb3f78dd9e49d5048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "375e2d8af3124e5cbd08b2fd878e8300" is_pre_election: true
I20260620 07:10:46.696288 17095 raft_consensus.cc:2454] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8474cedcfdc54d9bb3f78dd9e49d5048 in term 0.
I20260620 07:10:46.696347 16828 raft_consensus.cc:2454] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8474cedcfdc54d9bb3f78dd9e49d5048 in term 0.
I20260620 07:10:46.696979 16895 leader_election.cc:302] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39fe68012d2f47b3b54973f61505c4c0, 8474cedcfdc54d9bb3f78dd9e49d5048; no voters:
I20260620 07:10:46.697366 17164 raft_consensus.cc:2781] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:10:46.697511 17164 raft_consensus.cc:482] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:10:46.697621 17164 raft_consensus.cc:3037] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:46.699856 17164 raft_consensus.cc:504] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fe68012d2f47b3b54973f61505c4c0" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35549 } } peers { permanent_uuid: "375e2d8af3124e5cbd08b2fd878e8300" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33391 } } peers { permanent_uuid: "8474cedcfdc54d9bb3f78dd9e49d5048" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 41531 } }
I20260620 07:10:46.700461 17164 leader_election.cc:288] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [CANDIDATE]: Term 1 election: Requested vote from peers 39fe68012d2f47b3b54973f61505c4c0 (127.6.53.67:35549), 375e2d8af3124e5cbd08b2fd878e8300 (127.6.53.65:33391)
I20260620 07:10:46.700773 17095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9e9254dda449efbeb91d06fc720454" candidate_uuid: "8474cedcfdc54d9bb3f78dd9e49d5048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39fe68012d2f47b3b54973f61505c4c0"
I20260620 07:10:46.700963 17095 raft_consensus.cc:3037] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:46.700971 16828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc9e9254dda449efbeb91d06fc720454" candidate_uuid: "8474cedcfdc54d9bb3f78dd9e49d5048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "375e2d8af3124e5cbd08b2fd878e8300"
I20260620 07:10:46.701155 16828 raft_consensus.cc:3037] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:10:46.703168 17095 raft_consensus.cc:2454] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8474cedcfdc54d9bb3f78dd9e49d5048 in term 1.
I20260620 07:10:46.703364 16828 raft_consensus.cc:2454] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8474cedcfdc54d9bb3f78dd9e49d5048 in term 1.
I20260620 07:10:46.703539 16895 leader_election.cc:302] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39fe68012d2f47b3b54973f61505c4c0, 8474cedcfdc54d9bb3f78dd9e49d5048; no voters:
I20260620 07:10:46.703907 17164 raft_consensus.cc:2781] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:10:46.704293 17164 raft_consensus.cc:686] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [term 1 LEADER]: Becoming Leader. State: Replica: 8474cedcfdc54d9bb3f78dd9e49d5048, State: Running, Role: LEADER
I20260620 07:10:46.704735 17164 consensus_queue.cc:237] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fe68012d2f47b3b54973f61505c4c0" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35549 } } peers { permanent_uuid: "375e2d8af3124e5cbd08b2fd878e8300" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33391 } } peers { permanent_uuid: "8474cedcfdc54d9bb3f78dd9e49d5048" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 41531 } }
I20260620 07:10:46.709590 16684 catalog_manager.cc:5495] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8474cedcfdc54d9bb3f78dd9e49d5048 (127.6.53.66). New cstate: current_term: 1 leader_uuid: "8474cedcfdc54d9bb3f78dd9e49d5048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fe68012d2f47b3b54973f61505c4c0" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "375e2d8af3124e5cbd08b2fd878e8300" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8474cedcfdc54d9bb3f78dd9e49d5048" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 41531 } health_report { overall_health: HEALTHY } } }
I20260620 07:10:46.746757 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.742039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.6.53.66:41531)
0620 07:10:46.742289 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:46.742296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:46.742320 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:10:46.742431 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:46.742435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:46.742513 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:46.742645 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:10:46.742929 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.743632 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.744858 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.745219 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.745305 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.745708 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:10:46.746289 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:10:46.746389 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:46.746487 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 07:10:46.746546 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 07:10:46.754650 17095 raft_consensus.cc:1261] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0 [term 1 FOLLOWER]: Refusing update from remote peer 8474cedcfdc54d9bb3f78dd9e49d5048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:10:46.754649 16828 raft_consensus.cc:1261] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300 [term 1 FOLLOWER]: Refusing update from remote peer 8474cedcfdc54d9bb3f78dd9e49d5048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:10:46.755605 17169 consensus_queue.cc:1035] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39fe68012d2f47b3b54973f61505c4c0" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:46.755959 17164 consensus_queue.cc:1035] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375e2d8af3124e5cbd08b2fd878e8300" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 33391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:10:46.768751 17172 mvcc.cc:205] Tried to move back new op lower bound from 7298823973897084928 to 7298823973707108352. Current Snapshot: MvccSnapshot[applied={T|T < 7298823973897084928}]
I20260620 07:10:46.782559 16684 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:10:46.782842 16684 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:10:46.786391 16684 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ed9445c768c48bebeb21689a1c12d8d: Sending DeleteTablet for 3 replicas of tablet fc9e9254dda449efbeb91d06fc720454
I20260620 07:10:46.787451 16808 tablet_service.cc:1514] Processing DeleteTablet for tablet fc9e9254dda449efbeb91d06fc720454 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:46 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:45474
I20260620 07:10:46.787719 16942 tablet_service.cc:1514] Processing DeleteTablet for tablet fc9e9254dda449efbeb91d06fc720454 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:46 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:48392
I20260620 07:10:46.788127 17075 tablet_service.cc:1514] Processing DeleteTablet for tablet fc9e9254dda449efbeb91d06fc720454 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:10:46 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:44264
I20260620 07:10:46.789091 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 16745
I20260620 07:10:46.813000 17184 tablet_replica.cc:330] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0: stopping tablet replica
I20260620 07:10:46.813604 17184 raft_consensus.cc:2229] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:10:46.814262 17184 raft_consensus.cc:2258] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:46.815934 17182 tablet_replica.cc:330] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300: stopping tablet replica
I20260620 07:10:46.816339 17182 raft_consensus.cc:2229] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:10:46.816771 17182 raft_consensus.cc:2258] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:46.819703 17184 ts_tablet_manager.cc:1905] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:46.824044 17182 ts_tablet_manager.cc:1905] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 07:10:46 dist-test-slave-4n6h krb5kdc[16639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:10:46.830567 17183 tablet_replica.cc:330] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048: stopping tablet replica
I20260620 07:10:46.831034 17183 raft_consensus.cc:2229] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:10:46.831748 17183 raft_consensus.cc:2258] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:10:46.833451 17183 ts_tablet_manager.cc:1905] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:10:46.835021 17184 ts_tablet_manager.cc:1918] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:10:46.835196 17184 log.cc:1196] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/fc9e9254dda449efbeb91d06fc720454
I20260620 07:10:46.835736 17184 ts_tablet_manager.cc:1939] T fc9e9254dda449efbeb91d06fc720454 P 39fe68012d2f47b3b54973f61505c4c0: Deleting consensus metadata
I20260620 07:10:46.835741 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.790841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45478 (local address 127.6.53.65:33391)
0620 07:10:46.791055 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:46.791064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:46.815103 (+ 24039us) server_negotiation.cc:407] Connection header received
0620 07:10:46.815341 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:46.815346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:46.815453 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:46.815570 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:46.818300 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.819389 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.821193 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.821833 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.821936 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.827294 (+ 5358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:46.827306 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:46.827321 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:46.827361 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:46.831450 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:46.832788 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:46.832796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:46.832801 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:46.832960 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:46.833413 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:46.833417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:46.833420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:46.833918 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:46.834207 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:46.835001 (+ 794us) server_negotiation.cc:299] Negotiation successful
0620 07:10:46.835364 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:10:46.837512 16868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:46.837776 16668 catalog_manager.cc:4841] TS 39fe68012d2f47b3b54973f61505c4c0 (127.6.53.67:35549): tablet fc9e9254dda449efbeb91d06fc720454 (table test-table [id=fe5f724eac744f9fb5ddc830c0000018]) successfully deleted
I20260620 07:10:46.840747 17183 ts_tablet_manager.cc:1918] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:10:46.841022 17183 log.cc:1196] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/fc9e9254dda449efbeb91d06fc720454
I20260620 07:10:46.841837 17183 ts_tablet_manager.cc:1939] T fc9e9254dda449efbeb91d06fc720454 P 8474cedcfdc54d9bb3f78dd9e49d5048: Deleting consensus metadata
I20260620 07:10:46.843636 16668 catalog_manager.cc:4841] TS 8474cedcfdc54d9bb3f78dd9e49d5048 (127.6.53.66:41531): tablet fc9e9254dda449efbeb91d06fc720454 (table test-table [id=fe5f724eac744f9fb5ddc830c0000018]) successfully deleted
I20260620 07:10:46.926843 17182 ts_tablet_manager.cc:1918] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:10:46.927088 17182 log.cc:1196] T fc9e9254dda449efbeb91d06fc720454 P 375e2d8af3124e5cbd08b2fd878e8300: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/fc9e9254dda449efbeb91d06fc720454
I20260620 07:10:46.927493 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 16745
W20260620 07:10:46.945134 16670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.53.65:33391 (error 108)
I20260620 07:10:46.946350 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 16879
I20260620 07:10:46.947427 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.946894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:33391 (local address 127.0.0.1:45482)
0620 07:10:46.947161 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:46.947268 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260620 07:10:46.947669 16670 catalog_manager.cc:4584] TS 375e2d8af3124e5cbd08b2fd878e8300 (127.6.53.65:33391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc9e9254dda449efbeb91d06fc720454: Network error: Client connection negotiation failed: client connection to 127.6.53.65:33391: connect: Connection refused (error 111)
Jun 20 07:10:46 dist-test-slave-4n6h krb5kdc[16639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:10:46.961658 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:46.947250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48402 (local address 127.6.53.66:41531)
0620 07:10:46.947420 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:46.947427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:46.947447 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:10:46.947511 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:46.947517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:46.947593 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:46.947700 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:46.949065 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.949885 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.951315 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:46.951765 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:46.951901 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:46.955870 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:46.955885 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:46.955898 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:46.955929 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:46.958922 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:46.959557 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:46.959577 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:46.959579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:46.959706 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:46.960059 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:46.960066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:46.960069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:46.960480 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:46.960794 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:46.961053 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 07:10:46.961263 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:10:46.962415 17002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:47.001612 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:47.001030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:33391 (local address 127.0.0.1:45492)
0620 07:10:47.001293 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:47.001417 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 07:10:47.002732 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:47.002343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:33391 (local address 127.0.0.1:45498)
0620 07:10:47.002503 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:47.002589 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 07:10:47.063217 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:47.062674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:33391 (local address 127.0.0.1:45504)
0620 07:10:47.062901 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:47.063008 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 07:10:47.064292 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:47.063906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:33391 (local address 127.0.0.1:45520)
0620 07:10:47.064064 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:47.064157 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 07:10:47.079339 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 16879
I20260620 07:10:47.098392 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 17012
Jun 20 07:10:47 dist-test-slave-4n6h krb5kdc[16639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:10:47.112771 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:47.099128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.6.53.67:35549)
0620 07:10:47.099323 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:10:47.099327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:10:47.099346 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:10:47.099494 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:10:47.099498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:10:47.099559 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:10:47.099650 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:10:47.100773 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:47.101475 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:47.102728 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:10:47.103097 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:10:47.103436 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:10:47.107446 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:10:47.107460 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:10:47.107476 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:10:47.107515 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:10:47.110196 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:47.111005 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:47.111013 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:47.111016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:47.111153 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:10:47.111485 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:10:47.111488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:10:47.111490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:10:47.111831 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:10:47.112035 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 07:10:47.112365 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 07:10:47.112522 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:10:47.113366 17135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:47.145354 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:47.144881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:33391 (local address 127.0.0.1:45522)
0620 07:10:47.145109 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:47.145194 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 07:10:47.146440 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:10:47.146034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:33391 (local address 127.0.0.1:45524)
0620 07:10:47.146207 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 07:10:47.146339 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:33391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 07:10:47.201417 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 17012
I20260620 07:10:47.217861 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 16652
I20260620 07:10:47.219026 16714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:10:47.325899 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 16652
2026-06-20T07:10:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4169 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:10:47 dist-test-slave-4n6h krb5kdc[17193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:10:47 dist-test-slave-4n6h krb5kdc[17193](info): set up 2 sockets
Jun 20 07:10:47 dist-test-slave-4n6h krb5kdc[17193](info): commencing operation
krb5kdc: starting...
W20260620 07:10:48.415123 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:10:48 dist-test-slave-4n6h krb5kdc[17193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:10:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:10:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/rangeradmin_127.6.53.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/rangeradmin_127.6.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/rangerlookup_127.6.53.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/rangerlookup_127.6.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/HTTP_127.6.53.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/HTTP_127.6.53.84@KRBTEST.COM.keytab.
I20260620 07:10:48.709700 6357 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskS8B4yM/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/postgres -l logfile start
2026-06-20 07:10:51.206 UTC [17223] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 07:10:51.206 UTC [17223] LOG: listening on IPv4 address "127.6.53.84", port 50355
2026-06-20 07:10:51.210 UTC [17223] LOG: listening on Unix socket "/tmp/.s.PGSQL.50355"
2026-06-20 07:10:51.232 UTC [17226] LOG: database system was shut down at 2026-06-20 07:10:49 UTC
2026-06-20 07:10:51.235 UTC [17223] LOG: database system is ready to accept connections
I20260620 07:10:52.210933 6357 mini_postgres.cc:96] Postgres bound to 50355
2026-06-20 07:10:52.227 UTC [17234] FATAL: database "slave" does not exist
127.6.53.84:50355 - accepting connections
I20260620 07:10:52.228502 6357 mini_ranger.cc:161] Starting Ranger...
I20260620 07:10:52.253455 6357 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 07:10:52.741212 6357 mini_ranger.cc:87] Created ranger Postgres database
I20260620 07:10:52.741361 6357 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-admin
2026-06-20 07:10:53,033 [I] DB FLAVOR :POSTGRES
2026-06-20 07:10:53,034 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:10:53,034 [I] Checking connection..
2026-06-20 07:10:53,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 07:10:53,292 [I] Checking connection passed.
2026-06-20 07:10:53,293 [I] --------- Verifying version history table ---------
2026-06-20 07:10:53,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:10:53,540 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 07:10:53,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:10:53,785 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 07:10:53,785 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 07:10:53,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 07:10:54.029 UTC [17298] WARNING: there is no transaction in progress
2026-06-20 07:10:54,036 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 07:10:54,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:10:54,294 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 07:10:54,294 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:10:54,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 07:10:54,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 07:10:54,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4n6h.c.gcp-upstream.internal','N') ;"
2026-06-20 07:10:55,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 07:10:55,251 [I] Table x_portal_user does not exist in database ranger
2026-06-20 07:10:55,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 07:10:55,506 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 07:10:55,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 07:10:55,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 07:10:55,980 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 07:10:55,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 07:10:56.926 UTC [17406] WARNING: there is no transaction in progress
2026-06-20 07:10:56.941 UTC [17406] WARNING: there is no transaction in progress
2026-06-20 07:10:56.957 UTC [17406] WARNING: there is no transaction in progress
2026-06-20 07:10:56.972 UTC [17406] WARNING: there is no transaction in progress
2026-06-20 07:10:56.987 UTC [17406] WARNING: there is no transaction in progress
2026-06-20 07:10:57.004 UTC [17406] WARNING: there is no transaction in progress
2026-06-20 07:10:57.057 UTC [17406] WARNING: there is no transaction in progress
2026-06-20 07:10:57.064 UTC [17406] WARNING: there is no transaction in progress
2026-06-20 07:10:57.077 UTC [17406] WARNING: there is no transaction in progress
2026-06-20 07:10:57.087 UTC [17406] WARNING: there is no transaction in progress
2026-06-20 07:10:57.096 UTC [17406] WARNING: there is no transaction in progress
2026-06-20 07:10:57.110 UTC [17406] WARNING: there is no transaction in progress
2026-06-20 07:10:57.119 UTC [17406] WARNING: there is no transaction in progress
2026-06-20 07:10:57.125 UTC [17406] WARNING: there is no transaction in progress
2026-06-20 07:10:57.131 UTC [17406] WARNING: there is no transaction in progress
2026-06-20 07:10:57.137 UTC [17406] WARNING: there is no transaction in progress
2026-06-20 07:10:57,753 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 07:10:57,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 07:10:57,983 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 07:10:57,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 07:10:58,255 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 07:10:58,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 07:10:58,516 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 07:10:58,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 07:10:58,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 07:10:59,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4n6h.c.gcp-upstream.internal';"
2026-06-20 07:10:59,257 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 07:10:59,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 07:10:59,530 [I] DB_PATCHES have already been applied
I20260620 07:10:59.536136 6357 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-admin:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskS8B4yM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 07:10:59.536240 6357 mini_ranger.cc:193] Using host: 127.6.53.84
I20260620 07:10:59.550835 6357 mini_ranger.cc:227] Ranger admin URL: http://127.6.53.84:60987
Jun 20, 2026 7:10:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:10:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:10:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:10:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:10:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:11:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:11:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 07:11:00.680294 6357 mini_ranger.cc:160] Time spent starting Ranger: real 8.452s user 0.000s sys 0.037s
Jun 20, 2026 7:11:19 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:11:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 7:11:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:11:19 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:11:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:11:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:11:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:11:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:11:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:11:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:11:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:11:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:11:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:11:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:11:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:11:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:11:19 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:11:20 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 07:11:21.278796 6357 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/rangerkms_127.6.53.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/rangerkms_127.6.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.53.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.53.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.53.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/HTTP_127.6.53.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.53.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/HTTP_127.6.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 07:11:21 dist-test-slave-4n6h krb5kdc[17193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939481, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 07:11:21.428690 6357 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 07:11:21.447842 6357 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 07:11:22.232062 6357 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 07:11:22.232234 6357 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-kms
I20260620 07:11:22.232364 6357 mini_ranger_kms.cc:227] Using postgres at 127.6.53.84:50355
2026-06-20 07:11:22,655 [I] DB FLAVOR :POSTGRES
2026-06-20 07:11:22,655 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:11:22,655 [I] Checking connection
2026-06-20 07:11:22,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 07:11:22,883 [I] connection success
2026-06-20 07:11:22,883 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 07:11:22,883 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 07:11:22,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 07:11:23,154 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 07:11:23,155 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:11:23,155 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 07:11:23,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:50355/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 07:11:23,427 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 07:11:23.874836 6357 mini_ranger_kms.cc:310] Created kms service
I20260620 07:11:24.040333 6357 mini_ranger_kms.cc:326] Created kudu user
I20260620 07:11:24.083299 6357 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 07:11:24 dist-test-slave-4n6h krb5kdc[17193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939481, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
I20260620 07:11:24.451231 6357 mini_ranger_kms.cc:384] Added ranger policy
I20260620 07:11:24.451581 6357 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-kms:/tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 07:11:24.451643 6357 mini_ranger_kms.cc:240] Using host: 127.6.53.84
I20260620 07:11:24.469877 6357 mini_ranger_kms.cc:276] Ranger KMS PID: 17586
I20260620 07:11:24.470074 6357 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.53.84:57293
Jun 20, 2026 7:11:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:11:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:11:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:11:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:11:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:11:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 07:11:26 dist-test-slave-4n6h krb5kdc[17193](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939486, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.53.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:26.807122 6357 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.378s user 0.137s sys 0.300s
I20260620 07:11:26.807382 6357 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 07:11:26.807605 6357 mini_ranger_kms.cc:401] 127.6.53.84:57293/kms/v1/keys
Jun 20 07:11:27 dist-test-slave-4n6h krb5kdc[17193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939486, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.53.84@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
Jun 20 07:11:27 dist-test-slave-4n6h krb5kdc[17193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939486, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.53.84@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
Jun 20 07:11:29 dist-test-slave-4n6h krb5kdc[17193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:11:29.841630 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:35987
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:35987
--ranger_config_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.53.84:57293/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:30.178223 17637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:30.178521 17637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:30.178676 17637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:30.185734 17637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:11:30.185842 17637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:30.185950 17637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:30.186056 17637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:11:30.186138 17637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:11:30.196012 17637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.53.84:57293/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:35987
--ranger_config_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:35987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:11:30.197248 17637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:30.198549 17637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:30.210110 17642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:30.210155 17643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:30.210193 17645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:30.211112 17637 server_base.cc:1034] running on GCE node
I20260620 07:11:30.211799 17637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:30.213044 17637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:30.214233 17637 hybrid_clock.cc:648] HybridClock initialized: now 1781939490214196 us; error 28 us; skew 500 ppm
I20260620 07:11:30.214577 17637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:11:30 dist-test-slave-4n6h krb5kdc[17193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939490, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:30.221001 17637 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:11:30.222575 17637 webserver.cc:460] Webserver started at http://127.6.53.126:44533/ using document root <none> and password file <none>
I20260620 07:11:30.223152 17637 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:30.223242 17637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:30.223480 17637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:11:30 dist-test-slave-4n6h krb5kdc[17193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939490, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
I20260620 07:11:30.277608 17637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "107f769ef949403a99558b3e04945af8"
format_stamp: "Formatted at 2026-06-20 07:11:30 on dist-test-slave-4n6h"
server_key: "78a767a4589abba508fea33ad7c1323a"
server_key_iv: "af6d94b25fb95278705b07259107bdd6"
server_key_version: "kuduclusterkey@0"
I20260620 07:11:30.278520 17637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "107f769ef949403a99558b3e04945af8"
format_stamp: "Formatted at 2026-06-20 07:11:30 on dist-test-slave-4n6h"
server_key: "78a767a4589abba508fea33ad7c1323a"
server_key_iv: "af6d94b25fb95278705b07259107bdd6"
server_key_version: "kuduclusterkey@0"
I20260620 07:11:30.284812 17637 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 07:11:30.304421 17652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:30.305933 17637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 07:11:30.306113 17637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "107f769ef949403a99558b3e04945af8"
format_stamp: "Formatted at 2026-06-20 07:11:30 on dist-test-slave-4n6h"
server_key: "78a767a4589abba508fea33ad7c1323a"
server_key_iv: "af6d94b25fb95278705b07259107bdd6"
server_key_version: "kuduclusterkey@0"
I20260620 07:11:30.306414 17637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:30.320309 17637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:30.324379 17637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:30.324669 17637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:30.348205 17637 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:35987
I20260620 07:11:30.348237 17703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:35987 every 8 connection(s)
I20260620 07:11:30.349606 17637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:11:30.349977 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 17637
I20260620 07:11:30.350198 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:11:30.351087 6357 external_mini_cluster.cc:1428] Setting key 528d4d8e72b0918f22d48910fdeb1810
I20260620 07:11:30.354485 17704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 07:11:30 dist-test-slave-4n6h krb5kdc[17193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:11:30.369019 17704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8: Bootstrap starting.
I20260620 07:11:30.372380 17704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:30.373565 17704 log.cc:824] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:30.374454 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:30.353163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.6.53.126:35987)
0620 07:11:30.353973 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:30.353985 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:30.354024 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:11:30.355712 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:30.355732 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:30.356079 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:30.356397 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:30.358121 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.359574 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.361929 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.362735 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.362866 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:30.367674 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:30.367682 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:30.367697 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:30.367732 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:30.370647 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:30.371287 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:30.371300 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:30.371306 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:30.371451 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:30.371825 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:30.371831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:30.371834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:30.372295 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:30.372628 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:30.373045 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 07:11:30.373427 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":341,"threads_started":1}
I20260620 07:11:30.376577 17704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8: No bootstrap required, opened a new log
I20260620 07:11:30.385066 17704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107f769ef949403a99558b3e04945af8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35987 } }
I20260620 07:11:30.385432 17704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:30.385545 17704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 107f769ef949403a99558b3e04945af8, State: Initialized, Role: FOLLOWER
I20260620 07:11:30.386144 17704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107f769ef949403a99558b3e04945af8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35987 } }
I20260620 07:11:30.386458 17704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:11:30.386684 17704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:11:30.386881 17704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:30.389328 17704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107f769ef949403a99558b3e04945af8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35987 } }
I20260620 07:11:30.389873 17704 leader_election.cc:302] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 107f769ef949403a99558b3e04945af8; no voters:
I20260620 07:11:30.390506 17704 leader_election.cc:288] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:11:30.390594 17709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:30.391372 17709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [term 1 LEADER]: Becoming Leader. State: Replica: 107f769ef949403a99558b3e04945af8, State: Running, Role: LEADER
I20260620 07:11:30.391970 17709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107f769ef949403a99558b3e04945af8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35987 } }
I20260620 07:11:30.392736 17704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:11:30.395332 17710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "107f769ef949403a99558b3e04945af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107f769ef949403a99558b3e04945af8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35987 } } }
I20260620 07:11:30.395289 17711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 107f769ef949403a99558b3e04945af8. Latest consensus state: current_term: 1 leader_uuid: "107f769ef949403a99558b3e04945af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107f769ef949403a99558b3e04945af8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35987 } } }
I20260620 07:11:30.395864 17711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:30.395750 17710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:30.397094 17717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:11:30.403221 17717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:11:30.403973 17704 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 07:11:30.422003 17717 catalog_manager.cc:1347] Generated new cluster ID: 1a5f806a36b3474faa7cbc47ae007f30
I20260620 07:11:30.422142 17717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:11:30.441809 17717 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:11:30.444217 17717 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:11:30.454726 17717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8: Generated new TSK 0
I20260620 07:11:30.455830 17717 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:11:31.192267 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.53.84:57293/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:35987
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:11:31 dist-test-slave-4n6h krb5kdc[17193](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939491, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 07:11:31.632711 17749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:31.633105 17749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:31.633313 17749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:31.643389 17749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:31.643568 17749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:31.643779 17749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:11:31.661552 17749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.53.84:57293/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:35987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:11:31.663388 17749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:31.665251 17749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:31.685420 17759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:31.686879 17760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:31.700057 17762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:31.700433 17749 server_base.cc:1034] running on GCE node
I20260620 07:11:31.701272 17749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:31.702598 17749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:31.703949 17749 hybrid_clock.cc:648] HybridClock initialized: now 1781939491703842 us; error 93 us; skew 500 ppm
I20260620 07:11:31.704420 17749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:11:31 dist-test-slave-4n6h krb5kdc[17193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939491, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:31.709496 17749 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:11:31.711805 17749 webserver.cc:460] Webserver started at http://127.6.53.65:34121/ using document root <none> and password file <none>
I20260620 07:11:31.712844 17749 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:31.713057 17749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:31.713464 17749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:11:31 dist-test-slave-4n6h krb5kdc[17193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939491, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
I20260620 07:11:31.742274 17749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "ae0fc67d8b0d4787a80f45a6f802f9cb"
format_stamp: "Formatted at 2026-06-20 07:11:31 on dist-test-slave-4n6h"
server_key: "6e8b98a90015ee9c1cf6edb9ede7ccf0"
server_key_iv: "d6cbbc79fb3924cd2bcb84c4e8a99c9d"
server_key_version: "kuduclusterkey@0"
I20260620 07:11:31.743901 17749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "ae0fc67d8b0d4787a80f45a6f802f9cb"
format_stamp: "Formatted at 2026-06-20 07:11:31 on dist-test-slave-4n6h"
server_key: "6e8b98a90015ee9c1cf6edb9ede7ccf0"
server_key_iv: "d6cbbc79fb3924cd2bcb84c4e8a99c9d"
server_key_version: "kuduclusterkey@0"
I20260620 07:11:31.759124 17749 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.009s sys 0.004s
I20260620 07:11:31.777784 17769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:31.779714 17749 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 07:11:31.779948 17749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "ae0fc67d8b0d4787a80f45a6f802f9cb"
format_stamp: "Formatted at 2026-06-20 07:11:31 on dist-test-slave-4n6h"
server_key: "6e8b98a90015ee9c1cf6edb9ede7ccf0"
server_key_iv: "d6cbbc79fb3924cd2bcb84c4e8a99c9d"
server_key_version: "kuduclusterkey@0"
I20260620 07:11:31.780270 17749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:31.807081 17749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:31.812920 17749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:31.813345 17749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:31.820797 17749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:31.829051 17749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:31.829229 17749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 07:11:31.829463 17749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:31.829595 17749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:31.919166 17749 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:38591
I20260620 07:11:31.919248 17881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:38591 every 8 connection(s)
I20260620 07:11:31.920876 17749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:11:31.930224 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 17749
I20260620 07:11:31.930510 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:11:31.931557 6357 external_mini_cluster.cc:1428] Setting key 44a1b2832a3fc4b636dcc793c7cde6da
Jun 20 07:11:31 dist-test-slave-4n6h krb5kdc[17193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939491, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
I20260620 07:11:31.972663 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:31.929716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:35987 (local address 127.6.53.65:35337)
0620 07:11:31.938679 (+ 8963us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:31.938736 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:31.941967 (+ 3231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:31.945052 (+ 3085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:31.945065 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:31.945748 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:31.949383 (+ 3635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:31.949416 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:31.953994 (+ 4578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:31.954001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:31.956329 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:31.956346 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:31.958850 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:31.958859 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:31.959175 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:31.960109 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:31.960145 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:31.963241 (+ 3096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:31.967560 (+ 4319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:31.967570 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:31.967589 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:31.968237 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:31.968762 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:31.968767 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:31.968771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:31.969024 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:31.969735 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:31.969746 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:31.970221 (+ 475us) client_negotiation.cc:769] Sending connection context
0620 07:11:31.970577 (+ 356us) client_negotiation.cc:240] Negotiation successful
0620 07:11:31.971158 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8583,"thread_start_us":12437,"threads_started":1}
I20260620 07:11:31.973872 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:31.929152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:35337 (local address 127.6.53.126:35987)
0620 07:11:31.943429 (+ 14277us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:31.943438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:31.943463 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:11:31.944536 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:31.944542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:31.944663 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:31.944798 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:31.949673 (+ 4875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:31.953762 (+ 4089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:31.957643 (+ 3881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:31.958372 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:31.958509 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:31.963856 (+ 5347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:31.963862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:31.963868 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:31.963906 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:31.967392 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:31.968424 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:31.968430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:31.968434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:31.968596 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:31.969182 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:31.969188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:31.969191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:31.969551 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:31.969693 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:31.973201 (+ 3508us) server_negotiation.cc:299] Negotiation successful
0620 07:11:31.973439 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14115,"thread_start_us":13999,"threads_started":1}
I20260620 07:11:31.976293 17882 heartbeater.cc:347] Connected to a master server at 127.6.53.126:35987
I20260620 07:11:31.976668 17882 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:31.977591 17882 heartbeater.cc:511] Master 127.6.53.126:35987 requested a full tablet report, sending...
I20260620 07:11:31.980186 17669 ts_manager.cc:194] Registered new tserver with Master: ae0fc67d8b0d4787a80f45a6f802f9cb (127.6.53.65:38591)
I20260620 07:11:31.982318 17669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:35337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:11:32.013621 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.53.84:57293/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:35987
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:11:32 dist-test-slave-4n6h krb5kdc[17193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939491, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
Jun 20 07:11:32 dist-test-slave-4n6h krb5kdc[17193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939491, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
W20260620 07:11:32.510164 17888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:32.510682 17888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:32.510892 17888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:32.522962 17888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:32.523185 17888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:32.523442 17888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:11:32.540027 17888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.53.84:57293/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:35987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:11:32.541951 17888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:32.543833 17888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:32.591193 17895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:32.595269 17896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:32.605443 17888 server_base.cc:1034] running on GCE node
W20260620 07:11:32.606041 17899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:32.607102 17888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:32.613716 17888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:32.615074 17888 hybrid_clock.cc:648] HybridClock initialized: now 1781939492614952 us; error 121 us; skew 500 ppm
I20260620 07:11:32.615540 17888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:11:32 dist-test-slave-4n6h krb5kdc[17193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939492, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:32.619730 17888 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:11:32.637041 17888 webserver.cc:460] Webserver started at http://127.6.53.66:37977/ using document root <none> and password file <none>
I20260620 07:11:32.638047 17888 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:32.638240 17888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:32.638685 17888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:11:32 dist-test-slave-4n6h krb5kdc[17193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939492, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
I20260620 07:11:32.667873 17888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "8fc71d4e9ff645d0adbeb7527f1b3caf"
format_stamp: "Formatted at 2026-06-20 07:11:32 on dist-test-slave-4n6h"
server_key: "171c2f4b76bc754d2eafc1b9f2e87d8d"
server_key_iv: "8f34d95fe4fcd209ba037f34a471920b"
server_key_version: "kuduclusterkey@0"
I20260620 07:11:32.669278 17888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "8fc71d4e9ff645d0adbeb7527f1b3caf"
format_stamp: "Formatted at 2026-06-20 07:11:32 on dist-test-slave-4n6h"
server_key: "171c2f4b76bc754d2eafc1b9f2e87d8d"
server_key_iv: "8f34d95fe4fcd209ba037f34a471920b"
server_key_version: "kuduclusterkey@0"
I20260620 07:11:32.683940 17888 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.008s sys 0.000s
I20260620 07:11:32.711897 17912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:32.713840 17888 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.004s
I20260620 07:11:32.714099 17888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "8fc71d4e9ff645d0adbeb7527f1b3caf"
format_stamp: "Formatted at 2026-06-20 07:11:32 on dist-test-slave-4n6h"
server_key: "171c2f4b76bc754d2eafc1b9f2e87d8d"
server_key_iv: "8f34d95fe4fcd209ba037f34a471920b"
server_key_version: "kuduclusterkey@0"
I20260620 07:11:32.714462 17888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:32.759364 17888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:32.764595 17888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:32.764966 17888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:32.772409 17888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:32.780476 17888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:32.780640 17888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:32.780807 17888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:32.780901 17888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:32.986356 17882 heartbeater.cc:503] Master 127.6.53.126:35987 was elected leader, sending a full tablet report...
I20260620 07:11:33.213325 17888 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:43445
I20260620 07:11:33.213351 18024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:43445 every 8 connection(s)
I20260620 07:11:33.214931 17888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:11:33.224502 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 17888
I20260620 07:11:33.224690 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:11:33.225116 6357 external_mini_cluster.cc:1428] Setting key 3d3605615c965f670485eb93d8c257a7
Jun 20 07:11:33 dist-test-slave-4n6h krb5kdc[17193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939492, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:11:33.237509 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.217548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:52125 (local address 127.6.53.126:35987)
0620 07:11:33.218059 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:33.218065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:33.219087 (+ 1022us) server_negotiation.cc:407] Connection header received
0620 07:11:33.221613 (+ 2526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:33.221617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:33.221694 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:33.221809 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:33.223607 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.224719 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.226934 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.227481 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.227627 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.231090 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:33.231095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:33.231099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:33.231135 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:33.234153 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:33.234889 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:33.234895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:33.234901 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:33.235019 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:33.235482 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:33.235487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:33.235490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:33.235826 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:33.235969 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:33.236881 (+ 912us) server_negotiation.cc:299] Negotiation successful
0620 07:11:33.237130 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":236,"threads_started":1}
I20260620 07:11:33.238734 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.218034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:35987 (local address 127.6.53.66:52125)
0620 07:11:33.218892 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:33.218951 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:33.221298 (+ 2347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:33.222050 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:33.222060 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:33.222662 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:33.223429 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.223451 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.224876 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.224882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.226809 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.226824 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.227704 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.227713 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.227959 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.228679 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:33.228701 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:33.230871 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:33.234321 (+ 3450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:33.234330 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:33.234347 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:33.234762 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:33.235136 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:33.235143 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:33.235147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:33.235336 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:33.235966 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:33.235976 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:33.236430 (+ 454us) client_negotiation.cc:769] Sending connection context
0620 07:11:33.236744 (+ 314us) client_negotiation.cc:240] Negotiation successful
0620 07:11:33.237285 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":347,"threads_started":1}
I20260620 07:11:33.240430 18025 heartbeater.cc:347] Connected to a master server at 127.6.53.126:35987
I20260620 07:11:33.240732 18025 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:33.241529 18025 heartbeater.cc:511] Master 127.6.53.126:35987 requested a full tablet report, sending...
I20260620 07:11:33.243168 17669 ts_manager.cc:194] Registered new tserver with Master: 8fc71d4e9ff645d0adbeb7527f1b3caf (127.6.53.66:43445)
I20260620 07:11:33.244406 17669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:52125
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:11:33.279239 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.53.84:57293/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:35987
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:33.617332 18031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:33.617607 18031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:33.617766 18031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:33.625288 18031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:33.625392 18031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:33.625614 18031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:11:33.635795 18031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.53.84:57293/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:35987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:11:33.637058 18031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:33.638350 18031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:33.651715 18037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:33.651681 18039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:33.651875 18036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:33.652395 18031 server_base.cc:1034] running on GCE node
I20260620 07:11:33.652966 18031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:33.653935 18031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:33.655169 18031 hybrid_clock.cc:648] HybridClock initialized: now 1781939493655112 us; error 47 us; skew 500 ppm
I20260620 07:11:33.655478 18031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:11:33 dist-test-slave-4n6h krb5kdc[17193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939493, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:33.658490 18031 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:11:33.659904 18031 webserver.cc:460] Webserver started at http://127.6.53.67:32821/ using document root <none> and password file <none>
I20260620 07:11:33.660547 18031 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:33.660692 18031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:33.661003 18031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:11:33 dist-test-slave-4n6h krb5kdc[17193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939493, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
I20260620 07:11:33.676697 18031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "cb84d53c74f34adda4953fa71c309a58"
format_stamp: "Formatted at 2026-06-20 07:11:33 on dist-test-slave-4n6h"
server_key: "aaddeb96e2d2f7d06c7fa722ba4d4f81"
server_key_iv: "5f4eb01bb52841a2fbd653ffe4eae1ac"
server_key_version: "kuduclusterkey@0"
I20260620 07:11:33.677531 18031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "cb84d53c74f34adda4953fa71c309a58"
format_stamp: "Formatted at 2026-06-20 07:11:33 on dist-test-slave-4n6h"
server_key: "aaddeb96e2d2f7d06c7fa722ba4d4f81"
server_key_iv: "5f4eb01bb52841a2fbd653ffe4eae1ac"
server_key_version: "kuduclusterkey@0"
I20260620 07:11:33.683159 18031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 07:11:33.700939 18046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:33.702517 18031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:11:33.702688 18031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "cb84d53c74f34adda4953fa71c309a58"
format_stamp: "Formatted at 2026-06-20 07:11:33 on dist-test-slave-4n6h"
server_key: "aaddeb96e2d2f7d06c7fa722ba4d4f81"
server_key_iv: "5f4eb01bb52841a2fbd653ffe4eae1ac"
server_key_version: "kuduclusterkey@0"
I20260620 07:11:33.702906 18031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:33.717049 18031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:33.721237 18031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:33.721529 18031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:33.722623 18031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:33.724023 18031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:33.724094 18031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:33.724228 18031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:33.724303 18031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:33.760115 18031 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:35775
I20260620 07:11:33.760150 18158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:35775 every 8 connection(s)
I20260620 07:11:33.761353 18031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:11:33.761924 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 18031
I20260620 07:11:33.762099 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:11:33.762746 6357 external_mini_cluster.cc:1428] Setting key 80f7c1bcc8f8ddfa46558d08906765ab
Jun 20 07:11:33 dist-test-slave-4n6h krb5kdc[17193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939493, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:11:33.780778 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.764035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:60363 (local address 127.6.53.126:35987)
0620 07:11:33.764499 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:33.764505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:33.765973 (+ 1468us) server_negotiation.cc:407] Connection header received
0620 07:11:33.767854 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:33.767857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:33.767918 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:33.768003 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:33.769257 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.770012 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.771549 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.771971 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.772061 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.775190 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:33.775195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:33.775198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:33.775227 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:33.777828 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:33.778612 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:33.778622 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:33.778624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:33.778744 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:33.779170 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:33.779182 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:33.779185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:33.779434 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:33.779584 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:33.780201 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 07:11:33.780413 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":305,"threads_started":1}
I20260620 07:11:33.781602 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.764535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:35987 (local address 127.6.53.67:60363)
0620 07:11:33.765784 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:33.765843 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:33.767626 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:33.768189 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:33.768196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:33.768590 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:33.769127 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.769143 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.770157 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.770161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.771414 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.771425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.772141 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.772145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.772324 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.772902 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:33.772923 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:33.775017 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:33.777968 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:33.777976 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:33.777991 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:33.778490 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:33.778857 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:33.778863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:33.778867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:33.779053 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:33.779561 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:33.779569 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:33.779868 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 07:11:33.780110 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 07:11:33.780596 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":343,"threads_started":1}
I20260620 07:11:33.782920 18159 heartbeater.cc:347] Connected to a master server at 127.6.53.126:35987
I20260620 07:11:33.783222 18159 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:33.783907 18159 heartbeater.cc:511] Master 127.6.53.126:35987 requested a full tablet report, sending...
I20260620 07:11:33.785282 17669 ts_manager.cc:194] Registered new tserver with Master: cb84d53c74f34adda4953fa71c309a58 (127.6.53.67:35775)
I20260620 07:11:33.786530 17669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:60363
I20260620 07:11:33.791139 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:11:33.808553 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.795620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.6.53.126:35987)
0620 07:11:33.795811 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:33.795819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:33.796231 (+ 412us) server_negotiation.cc:407] Connection header received
0620 07:11:33.796358 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:33.796361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:33.796430 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:33.796505 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:33.797737 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.798480 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.800621 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.801027 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.801129 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.803598 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:33.803602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:33.803606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:33.803634 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:33.806198 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:33.806944 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:33.806948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:33.806950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:33.807060 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:33.807397 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:33.807401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:33.807404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:33.807704 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:33.807865 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:33.808162 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 07:11:33.808330 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:11:33.812960 17669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:11:33.820801 17669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:11:33.866231 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.833425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42382 (local address 127.6.53.65:38591)
0620 07:11:33.850828 (+ 17403us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:33.850842 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:33.855771 (+ 4929us) server_negotiation.cc:407] Connection header received
0620 07:11:33.855918 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:33.855930 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:33.856265 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:33.856506 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:33.857769 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.860026 (+ 2257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.863033 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.864988 (+ 1955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.865133 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.865439 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:33.865705 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 07:11:33.865876 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17190,"thread_start_us":17028,"threads_started":1}
I20260620 07:11:33.866281 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.833110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38591 (local address 127.0.0.1:42382)
0620 07:11:33.855137 (+ 22027us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:33.855214 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:33.855495 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:33.856871 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:33.856882 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:33.856923 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:33.857497 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.857513 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.860300 (+ 2787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.860308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.862858 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.862875 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.865239 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.865245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.865462 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.865508 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 07:11:33.865650 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 07:11:33.865838 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21684,"mutex_wait_us":1,"thread_start_us":27517,"threads_started":1}
I20260620 07:11:33.867825 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.833288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:35775 (local address 127.0.0.1:42170)
0620 07:11:33.859220 (+ 25932us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:33.859240 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:33.859362 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:33.860319 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:33.860323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:33.860346 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:33.860985 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.861001 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.862674 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.862682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.865192 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.865206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.867009 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.867015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.867206 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.867239 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:11:33.867386 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 07:11:33.867504 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25820,"thread_start_us":25679,"threads_started":1}
I20260620 07:11:33.868080 18175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.833756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.6.53.67:35775)
0620 07:11:33.858551 (+ 24795us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:33.858563 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:33.859493 (+ 930us) server_negotiation.cc:407] Connection header received
0620 07:11:33.859596 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:33.859604 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:33.859844 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:33.860023 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:33.861194 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.862551 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.865357 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.866803 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.866951 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.867172 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:33.867421 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 07:11:33.867569 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24578,"thread_start_us":24814,"threads_started":1}
I20260620 07:11:33.868960 18172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.836007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:43445 (local address 127.0.0.1:52522)
0620 07:11:33.860957 (+ 24950us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:33.860979 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:33.861106 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:33.861703 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:33.861708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:33.861735 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:33.862110 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.862121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.863768 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.863774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.865759 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.865771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.868241 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.868249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.868417 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.868447 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:11:33.868541 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 07:11:33.868656 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24791,"thread_start_us":18614,"threads_started":1}
I20260620 07:11:33.869047 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.840291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52522 (local address 127.6.53.66:43445)
0620 07:11:33.854152 (+ 13861us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:33.854165 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:33.861105 (+ 6940us) server_negotiation.cc:407] Connection header received
0620 07:11:33.861271 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:33.861281 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:33.861495 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:33.861720 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:33.862323 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.863564 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.865920 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.868028 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.868165 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.868335 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:33.868602 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 07:11:33.868758 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13651,"thread_start_us":13892,"threads_started":1}
I20260620 07:11:33.870204 17812 tablet_service.cc:1467] Processing CreateTablet for tablet 61a4dd66442544d0a91e69db577c172a (DEFAULT_TABLE table=test-table [id=c6f58d4aa67d467c9734e2f27ba03e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:33.871150 18094 tablet_service.cc:1467] Processing CreateTablet for tablet 61a4dd66442544d0a91e69db577c172a (DEFAULT_TABLE table=test-table [id=c6f58d4aa67d467c9734e2f27ba03e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:33.871874 17812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a4dd66442544d0a91e69db577c172a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:33.872303 18094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a4dd66442544d0a91e69db577c172a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:33.873574 17960 tablet_service.cc:1467] Processing CreateTablet for tablet 61a4dd66442544d0a91e69db577c172a (DEFAULT_TABLE table=test-table [id=c6f58d4aa67d467c9734e2f27ba03e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:33.874842 17960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a4dd66442544d0a91e69db577c172a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:33.883669 18178 tablet_bootstrap.cc:492] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58: Bootstrap starting.
I20260620 07:11:33.886787 18178 tablet_bootstrap.cc:654] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:33.886833 18179 tablet_bootstrap.cc:492] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb: Bootstrap starting.
I20260620 07:11:33.888411 18178 log.cc:824] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:33.890975 18180 tablet_bootstrap.cc:492] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf: Bootstrap starting.
I20260620 07:11:33.891119 18179 tablet_bootstrap.cc:654] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:33.892338 18179 log.cc:824] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:33.894470 18178 tablet_bootstrap.cc:492] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58: No bootstrap required, opened a new log
I20260620 07:11:33.894467 18180 tablet_bootstrap.cc:654] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:33.894747 18178 ts_tablet_manager.cc:1397] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260620 07:11:33.895056 18179 tablet_bootstrap.cc:492] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb: No bootstrap required, opened a new log
I20260620 07:11:33.895292 18179 ts_tablet_manager.cc:1397] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260620 07:11:33.895591 18180 log.cc:824] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:33.898816 18180 tablet_bootstrap.cc:492] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf: No bootstrap required, opened a new log
I20260620 07:11:33.899049 18180 ts_tablet_manager.cc:1397] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 07:11:33.903332 18178 raft_consensus.cc:348] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0fc67d8b0d4787a80f45a6f802f9cb" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38591 } } peers { permanent_uuid: "cb84d53c74f34adda4953fa71c309a58" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35775 } } peers { permanent_uuid: "8fc71d4e9ff645d0adbeb7527f1b3caf" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43445 } }
I20260620 07:11:33.903690 18178 raft_consensus.cc:374] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:33.903812 18178 raft_consensus.cc:729] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb84d53c74f34adda4953fa71c309a58, State: Initialized, Role: FOLLOWER
I20260620 07:11:33.904412 18178 consensus_queue.cc:260] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0fc67d8b0d4787a80f45a6f802f9cb" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38591 } } peers { permanent_uuid: "cb84d53c74f34adda4953fa71c309a58" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35775 } } peers { permanent_uuid: "8fc71d4e9ff645d0adbeb7527f1b3caf" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43445 } }
I20260620 07:11:33.905462 18159 heartbeater.cc:503] Master 127.6.53.126:35987 was elected leader, sending a full tablet report...
I20260620 07:11:33.906106 18178 ts_tablet_manager.cc:1428] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 07:11:33.905874 18179 raft_consensus.cc:348] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0fc67d8b0d4787a80f45a6f802f9cb" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38591 } } peers { permanent_uuid: "cb84d53c74f34adda4953fa71c309a58" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35775 } } peers { permanent_uuid: "8fc71d4e9ff645d0adbeb7527f1b3caf" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43445 } }
I20260620 07:11:33.906301 18179 raft_consensus.cc:374] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:33.906410 18179 raft_consensus.cc:729] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0fc67d8b0d4787a80f45a6f802f9cb, State: Initialized, Role: FOLLOWER
I20260620 07:11:33.906970 18179 consensus_queue.cc:260] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0fc67d8b0d4787a80f45a6f802f9cb" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38591 } } peers { permanent_uuid: "cb84d53c74f34adda4953fa71c309a58" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35775 } } peers { permanent_uuid: "8fc71d4e9ff645d0adbeb7527f1b3caf" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43445 } }
I20260620 07:11:33.908763 18179 ts_tablet_manager.cc:1428] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb: Time spent starting tablet: real 0.013s user 0.004s sys 0.010s
I20260620 07:11:33.909526 18180 raft_consensus.cc:348] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0fc67d8b0d4787a80f45a6f802f9cb" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38591 } } peers { permanent_uuid: "cb84d53c74f34adda4953fa71c309a58" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35775 } } peers { permanent_uuid: "8fc71d4e9ff645d0adbeb7527f1b3caf" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43445 } }
I20260620 07:11:33.909920 18180 raft_consensus.cc:374] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:33.910022 18180 raft_consensus.cc:729] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fc71d4e9ff645d0adbeb7527f1b3caf, State: Initialized, Role: FOLLOWER
I20260620 07:11:33.910549 18180 consensus_queue.cc:260] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0fc67d8b0d4787a80f45a6f802f9cb" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38591 } } peers { permanent_uuid: "cb84d53c74f34adda4953fa71c309a58" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35775 } } peers { permanent_uuid: "8fc71d4e9ff645d0adbeb7527f1b3caf" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43445 } }
I20260620 07:11:33.911689 18025 heartbeater.cc:503] Master 127.6.53.126:35987 was elected leader, sending a full tablet report...
I20260620 07:11:33.912194 18180 ts_tablet_manager.cc:1428] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
W20260620 07:11:33.946022 17883 tablet.cc:2369] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:11:33.967435 18026 tablet.cc:2369] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:11:34.013078 18160 tablet.cc:2369] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:11:34.164755 18184 raft_consensus.cc:482] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:11:34.165531 18184 raft_consensus.cc:504] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0fc67d8b0d4787a80f45a6f802f9cb" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38591 } } peers { permanent_uuid: "cb84d53c74f34adda4953fa71c309a58" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35775 } } peers { permanent_uuid: "8fc71d4e9ff645d0adbeb7527f1b3caf" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43445 } }
I20260620 07:11:34.167963 18184 leader_election.cc:288] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0fc67d8b0d4787a80f45a6f802f9cb (127.6.53.65:38591), 8fc71d4e9ff645d0adbeb7527f1b3caf (127.6.53.66:43445)
I20260620 07:11:34.175634 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:34.169240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:33961 (local address 127.6.53.65:38591)
0620 07:11:34.169672 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:34.169677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:34.169694 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:11:34.169756 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:34.169760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:34.169815 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:34.169933 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:34.170582 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.171537 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.173900 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.174742 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.174896 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:34.175001 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:34.175265 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 07:11:34.175393 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 07:11:34.175653 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:34.168317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38591 (local address 127.6.53.67:33961)
0620 07:11:34.168469 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:34.168493 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:34.168632 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:34.169948 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:34.169952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:34.170050 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:34.170399 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:34.170417 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.171675 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.171683 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:34.173723 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:34.173736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.174904 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.174910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:34.175107 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:34.175144 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 07:11:34.175240 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 07:11:34.175369 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":65,"spinlock_wait_cycles":21376}
I20260620 07:11:34.176877 17837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a4dd66442544d0a91e69db577c172a" candidate_uuid: "cb84d53c74f34adda4953fa71c309a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0fc67d8b0d4787a80f45a6f802f9cb" is_pre_election: true
I20260620 07:11:34.177363 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:34.169337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:43077 (local address 127.6.53.66:43445)
0620 07:11:34.169532 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:34.169539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:34.169558 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:11:34.169629 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:34.169634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:34.169740 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:34.169849 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:34.170541 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.171820 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.174851 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.176197 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.176277 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:34.176392 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:34.176987 (+ 595us) server_negotiation.cc:299] Negotiation successful
0620 07:11:34.177128 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:11:34.177475 17837 raft_consensus.cc:2454] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb84d53c74f34adda4953fa71c309a58 in term 0.
I20260620 07:11:34.178037 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:34.168617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:43445 (local address 127.6.53.67:43077)
0620 07:11:34.169225 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:34.169242 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:34.169365 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:34.169883 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:34.169888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:34.169912 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:34.170370 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:34.170387 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.171960 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.171965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:34.174343 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:34.174358 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.176629 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.176635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:34.176801 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:34.176830 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 07:11:34.177534 (+ 704us) client_negotiation.cc:240] Negotiation successful
0620 07:11:34.177675 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":510,"threads_started":1}
I20260620 07:11:34.178223 18048 leader_election.cc:302] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0fc67d8b0d4787a80f45a6f802f9cb, cb84d53c74f34adda4953fa71c309a58; no voters:
I20260620 07:11:34.178710 18184 raft_consensus.cc:2781] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:11:34.178875 18184 raft_consensus.cc:482] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:11:34.178985 18184 raft_consensus.cc:3037] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:34.179207 17980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a4dd66442544d0a91e69db577c172a" candidate_uuid: "cb84d53c74f34adda4953fa71c309a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fc71d4e9ff645d0adbeb7527f1b3caf" is_pre_election: true
I20260620 07:11:34.179729 17980 raft_consensus.cc:2454] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb84d53c74f34adda4953fa71c309a58 in term 0.
I20260620 07:11:34.182337 18184 raft_consensus.cc:504] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0fc67d8b0d4787a80f45a6f802f9cb" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38591 } } peers { permanent_uuid: "cb84d53c74f34adda4953fa71c309a58" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35775 } } peers { permanent_uuid: "8fc71d4e9ff645d0adbeb7527f1b3caf" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43445 } }
I20260620 07:11:34.183209 18184 leader_election.cc:288] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [CANDIDATE]: Term 1 election: Requested vote from peers ae0fc67d8b0d4787a80f45a6f802f9cb (127.6.53.65:38591), 8fc71d4e9ff645d0adbeb7527f1b3caf (127.6.53.66:43445)
I20260620 07:11:34.183653 17837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a4dd66442544d0a91e69db577c172a" candidate_uuid: "cb84d53c74f34adda4953fa71c309a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0fc67d8b0d4787a80f45a6f802f9cb"
I20260620 07:11:34.183894 17837 raft_consensus.cc:3037] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:34.186499 17980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a4dd66442544d0a91e69db577c172a" candidate_uuid: "cb84d53c74f34adda4953fa71c309a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fc71d4e9ff645d0adbeb7527f1b3caf"
I20260620 07:11:34.186560 17837 raft_consensus.cc:2454] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb84d53c74f34adda4953fa71c309a58 in term 1.
I20260620 07:11:34.186728 17980 raft_consensus.cc:3037] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:34.186945 18048 leader_election.cc:302] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae0fc67d8b0d4787a80f45a6f802f9cb, cb84d53c74f34adda4953fa71c309a58; no voters:
I20260620 07:11:34.187247 18184 raft_consensus.cc:2781] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:34.187747 18184 raft_consensus.cc:686] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [term 1 LEADER]: Becoming Leader. State: Replica: cb84d53c74f34adda4953fa71c309a58, State: Running, Role: LEADER
I20260620 07:11:34.188208 18184 consensus_queue.cc:237] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0fc67d8b0d4787a80f45a6f802f9cb" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38591 } } peers { permanent_uuid: "cb84d53c74f34adda4953fa71c309a58" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35775 } } peers { permanent_uuid: "8fc71d4e9ff645d0adbeb7527f1b3caf" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43445 } }
I20260620 07:11:34.189399 17980 raft_consensus.cc:2454] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb84d53c74f34adda4953fa71c309a58 in term 1.
I20260620 07:11:34.192379 17668 catalog_manager.cc:5495] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb84d53c74f34adda4953fa71c309a58 (127.6.53.67). New cstate: current_term: 1 leader_uuid: "cb84d53c74f34adda4953fa71c309a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0fc67d8b0d4787a80f45a6f802f9cb" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb84d53c74f34adda4953fa71c309a58" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fc71d4e9ff645d0adbeb7527f1b3caf" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43445 } health_report { overall_health: UNKNOWN } } }
I20260620 07:11:34.236968 18175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:34.232208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42178 (local address 127.6.53.67:35775)
0620 07:11:34.232375 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:34.232379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:34.232396 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:11:34.232476 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:34.232480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:34.232557 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:34.232702 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:11:34.233016 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.233680 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.235074 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.235468 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.235564 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:34.236018 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:11:34.236486 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:11:34.236588 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:34.236713 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 07:11:34.236775 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:11:34.244704 17980 raft_consensus.cc:1261] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf [term 1 FOLLOWER]: Refusing update from remote peer cb84d53c74f34adda4953fa71c309a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:34.244701 17837 raft_consensus.cc:1261] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb [term 1 FOLLOWER]: Refusing update from remote peer cb84d53c74f34adda4953fa71c309a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:34.245669 18188 consensus_queue.cc:1035] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0fc67d8b0d4787a80f45a6f802f9cb" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:34.246021 18184 consensus_queue.cc:1035] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fc71d4e9ff645d0adbeb7527f1b3caf" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:34.258136 18191 mvcc.cc:205] Tried to move back new op lower bound from 7298824168416448512 to 7298824168198438912. Current Snapshot: MvccSnapshot[applied={T|T < 7298824168416448512}]
I20260620 07:11:34.261394 18193 mvcc.cc:205] Tried to move back new op lower bound from 7298824168416448512 to 7298824168198438912. Current Snapshot: MvccSnapshot[applied={T|T < 7298824168416448512}]
I20260620 07:11:34.272356 17668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:34.272629 17668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:34.276239 17668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 107f769ef949403a99558b3e04945af8: Sending DeleteTablet for 3 replicas of tablet 61a4dd66442544d0a91e69db577c172a
I20260620 07:11:34.277369 18094 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a4dd66442544d0a91e69db577c172a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:34 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:42170
I20260620 07:11:34.277385 17812 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a4dd66442544d0a91e69db577c172a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:34 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:42382
I20260620 07:11:34.277480 17960 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a4dd66442544d0a91e69db577c172a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:34 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:52522
I20260620 07:11:34.278585 18201 tablet_replica.cc:330] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58: stopping tablet replica
I20260620 07:11:34.279091 18202 tablet_replica.cc:330] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb: stopping tablet replica
I20260620 07:11:34.279558 18202 raft_consensus.cc:2229] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:34.279822 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 17749
I20260620 07:11:34.280025 18201 raft_consensus.cc:2229] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:11:34.280647 18201 raft_consensus.cc:2258] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:34.281267 18202 raft_consensus.cc:2258] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:34.283869 18202 ts_tablet_manager.cc:1905] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:34.286940 18203 tablet_replica.cc:330] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf: stopping tablet replica
I20260620 07:11:34.287333 18203 raft_consensus.cc:2229] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:34.287802 18203 raft_consensus.cc:2258] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:34.288728 18201 ts_tablet_manager.cc:1905] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:34.289651 18203 ts_tablet_manager.cc:1905] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:34.296587 18202 ts_tablet_manager.cc:1918] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:11:34.296757 18202 log.cc:1196] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/61a4dd66442544d0a91e69db577c172a
Jun 20 07:11:34 dist-test-slave-4n6h krb5kdc[17193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:11:34.297292 18202 ts_tablet_manager.cc:1939] T 61a4dd66442544d0a91e69db577c172a P ae0fc67d8b0d4787a80f45a6f802f9cb: Deleting consensus metadata
I20260620 07:11:34.297272 18201 ts_tablet_manager.cc:1918] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:11:34.297555 18201 log.cc:1196] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/61a4dd66442544d0a91e69db577c172a
I20260620 07:11:34.298130 18201 ts_tablet_manager.cc:1939] T 61a4dd66442544d0a91e69db577c172a P cb84d53c74f34adda4953fa71c309a58: Deleting consensus metadata
I20260620 07:11:34.299242 17654 catalog_manager.cc:4841] TS ae0fc67d8b0d4787a80f45a6f802f9cb (127.6.53.65:38591): tablet 61a4dd66442544d0a91e69db577c172a (table test-table [id=c6f58d4aa67d467c9734e2f27ba03e72]) successfully deleted
I20260620 07:11:34.303040 17656 catalog_manager.cc:4841] TS cb84d53c74f34adda4953fa71c309a58 (127.6.53.67:35775): tablet 61a4dd66442544d0a91e69db577c172a (table test-table [id=c6f58d4aa67d467c9734e2f27ba03e72]) successfully deleted
I20260620 07:11:34.304067 18203 ts_tablet_manager.cc:1918] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:11:34.304239 18203 log.cc:1196] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/61a4dd66442544d0a91e69db577c172a
I20260620 07:11:34.304715 18203 ts_tablet_manager.cc:1939] T 61a4dd66442544d0a91e69db577c172a P 8fc71d4e9ff645d0adbeb7527f1b3caf: Deleting consensus metadata
I20260620 07:11:34.305797 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:34.282364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42388 (local address 127.6.53.65:38591)
0620 07:11:34.282609 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:34.282618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:34.286929 (+ 4311us) server_negotiation.cc:407] Connection header received
0620 07:11:34.287000 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:34.287008 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:34.287095 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:34.287196 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:34.289044 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.290112 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.292060 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.292763 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.292860 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:34.298465 (+ 5605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:34.298482 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:34.298498 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:34.298541 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:34.302349 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:34.303234 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:34.303243 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:34.303247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:34.303458 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:34.303985 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:34.303990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:34.303994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:34.304446 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:34.304774 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:34.305191 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 07:11:34.305411 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 07:11:34.306561 17877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:11:34.306658 17653 catalog_manager.cc:4841] TS 8fc71d4e9ff645d0adbeb7527f1b3caf (127.6.53.66:43445): tablet 61a4dd66442544d0a91e69db577c172a (table test-table [id=c6f58d4aa67d467c9734e2f27ba03e72]) successfully deleted
I20260620 07:11:34.413820 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 17749
I20260620 07:11:34.432305 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 17888
Jun 20 07:11:34 dist-test-slave-4n6h krb5kdc[17193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:11:34.449419 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:34.433143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.6.53.66:43445)
0620 07:11:34.433337 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:34.433341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:34.433362 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:11:34.433593 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:34.433597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:34.433678 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:34.433770 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:34.435430 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.436154 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.437994 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.438468 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.438560 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:34.443503 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:34.443517 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:34.443532 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:34.443565 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:34.446615 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:34.447491 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:34.447497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:34.447499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:34.447624 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:34.448038 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:34.448041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:34.448043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:34.448388 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:34.448593 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:34.448937 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 07:11:34.449096 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:11:34.450073 18020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:11:34.540115 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 17888
I20260620 07:11:34.557720 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 18031
Jun 20 07:11:34 dist-test-slave-4n6h krb5kdc[17193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:11:34.574764 18175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:34.558611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.6.53.67:35775)
0620 07:11:34.558830 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:34.558835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:34.558855 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:11:34.559027 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:34.559033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:34.559104 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:34.559391 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:34.560594 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.561416 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.562709 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.563157 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.563315 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:34.568615 (+ 5300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:34.568668 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:34.568688 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:34.568728 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:34.571920 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:34.572637 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:34.572645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:34.572647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:34.572778 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:34.573166 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:34.573172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:34.573177 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:34.573541 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:34.573877 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:34.574163 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 07:11:34.574411 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 07:11:34.575518 18154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:11:34.704005 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 18031
I20260620 07:11:34.723507 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 17637
I20260620 07:11:34.724632 17699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:11:34.834225 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 17637
I20260620 07:11:34.861270 6357 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 07:11:35.196112 6357 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 07:11:35.196350 6357 mini_ranger.cc:66] Stopping Ranger...
I20260620 07:11:35.541477 6357 mini_ranger.cc:68] Stopped Ranger
2026-06-20 07:11:35.541 UTC [17223] LOG: received smart shutdown request
2026-06-20 07:11:35.544 UTC [17223] LOG: background worker "logical replication launcher" (PID 17232) exited with exit code 1
2026-06-20 07:11:35.545 UTC [17227] LOG: shutting down
2026-06-20 07:11:35.703 UTC [17223] LOG: database system is shut down
2026-06-20T07:11:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48511 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:11:35 dist-test-slave-4n6h krb5kdc[18217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:11:35 dist-test-slave-4n6h krb5kdc[18217](info): set up 2 sockets
Jun 20 07:11:35 dist-test-slave-4n6h krb5kdc[18217](info): commencing operation
krb5kdc: starting...
W20260620 07:11:36.970520 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:11:37 dist-test-slave-4n6h krb5kdc[18217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:11:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:11:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/rangeradmin_127.6.53.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/rangeradmin_127.6.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/rangerlookup_127.6.53.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/rangerlookup_127.6.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/HTTP_127.6.53.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/HTTP_127.6.53.84@KRBTEST.COM.keytab.
I20260620 07:11:37.277091 6357 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskS8B4yM/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/postgres -l logfile start
2026-06-20 07:11:39.609 UTC [18247] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 07:11:39.609 UTC [18247] LOG: listening on IPv4 address "127.6.53.84", port 46643
2026-06-20 07:11:39.612 UTC [18247] LOG: listening on Unix socket "/tmp/.s.PGSQL.46643"
2026-06-20 07:11:39.632 UTC [18250] LOG: database system was shut down at 2026-06-20 07:11:38 UTC
2026-06-20 07:11:39.635 UTC [18247] LOG: database system is ready to accept connections
I20260620 07:11:40.610502 6357 mini_postgres.cc:96] Postgres bound to 46643
2026-06-20 07:11:40.627 UTC [18258] FATAL: database "slave" does not exist
127.6.53.84:46643 - accepting connections
I20260620 07:11:40.628549 6357 mini_ranger.cc:161] Starting Ranger...
I20260620 07:11:40.653563 6357 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 07:11:41.066325 6357 mini_ranger.cc:87] Created ranger Postgres database
I20260620 07:11:41.066439 6357 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-admin
2026-06-20 07:11:41,351 [I] DB FLAVOR :POSTGRES
2026-06-20 07:11:41,351 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:11:41,351 [I] Checking connection..
2026-06-20 07:11:41,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 07:11:41,614 [I] Checking connection passed.
2026-06-20 07:11:41,614 [I] --------- Verifying version history table ---------
2026-06-20 07:11:41,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:11:41,875 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 07:11:41,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:11:42,136 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 07:11:42,136 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 07:11:42,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 07:11:42.358 UTC [18322] WARNING: there is no transaction in progress
2026-06-20 07:11:42,365 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 07:11:42,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:11:42,606 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 07:11:42,606 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:11:42,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 07:11:42,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 07:11:43,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4n6h.c.gcp-upstream.internal','N') ;"
2026-06-20 07:11:43,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 07:11:43,540 [I] Table x_portal_user does not exist in database ranger
2026-06-20 07:11:43,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 07:11:43,773 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 07:11:43,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 07:11:44,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 07:11:44,256 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 07:11:44,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 07:11:45.173 UTC [18430] WARNING: there is no transaction in progress
2026-06-20 07:11:45.190 UTC [18430] WARNING: there is no transaction in progress
2026-06-20 07:11:45.206 UTC [18430] WARNING: there is no transaction in progress
2026-06-20 07:11:45.220 UTC [18430] WARNING: there is no transaction in progress
2026-06-20 07:11:45.238 UTC [18430] WARNING: there is no transaction in progress
2026-06-20 07:11:45.256 UTC [18430] WARNING: there is no transaction in progress
2026-06-20 07:11:45.313 UTC [18430] WARNING: there is no transaction in progress
2026-06-20 07:11:45.319 UTC [18430] WARNING: there is no transaction in progress
2026-06-20 07:11:45.333 UTC [18430] WARNING: there is no transaction in progress
2026-06-20 07:11:45.343 UTC [18430] WARNING: there is no transaction in progress
2026-06-20 07:11:45.354 UTC [18430] WARNING: there is no transaction in progress
2026-06-20 07:11:45.367 UTC [18430] WARNING: there is no transaction in progress
2026-06-20 07:11:45.376 UTC [18430] WARNING: there is no transaction in progress
2026-06-20 07:11:45.382 UTC [18430] WARNING: there is no transaction in progress
2026-06-20 07:11:45.388 UTC [18430] WARNING: there is no transaction in progress
2026-06-20 07:11:45.394 UTC [18430] WARNING: there is no transaction in progress
2026-06-20 07:11:45,989 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 07:11:45,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 07:11:46,228 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 07:11:46,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 07:11:46,466 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 07:11:46,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 07:11:46,722 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 07:11:46,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 07:11:46,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 07:11:47,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4n6h.c.gcp-upstream.internal';"
2026-06-20 07:11:47,406 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 07:11:47,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 07:11:47,656 [I] DB_PATCHES have already been applied
I20260620 07:11:47.662739 6357 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-admin:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskS8B4yM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 07:11:47.662814 6357 mini_ranger.cc:193] Using host: 127.6.53.84
I20260620 07:11:47.676630 6357 mini_ranger.cc:227] Ranger admin URL: http://127.6.53.84:58967
Jun 20, 2026 7:11:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:11:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:11:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:11:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:11:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:11:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:11:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 07:11:48.863387 6357 mini_ranger.cc:160] Time spent starting Ranger: real 8.235s user 0.004s sys 0.032s
Jun 20, 2026 7:12:08 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:12:08 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 7:12:08 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:12:08 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:12:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:12:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:12:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:12:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:12:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:12:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:12:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:12:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:12:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:12:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:12:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:12:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:12:08 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:12:09 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 07:12:10.280561 6357 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/rangerkms_127.6.53.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/rangerkms_127.6.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.53.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.53.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.53.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/HTTP_127.6.53.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.53.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/HTTP_127.6.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 07:12:10 dist-test-slave-4n6h krb5kdc[18217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939530, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 07:12:10.448725 6357 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 07:12:10.472654 6357 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 07:12:11.295353 6357 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 07:12:11.295450 6357 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-kms
I20260620 07:12:11.295481 6357 mini_ranger_kms.cc:227] Using postgres at 127.6.53.84:46643
2026-06-20 07:12:11,634 [I] DB FLAVOR :POSTGRES
2026-06-20 07:12:11,635 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:12:11,635 [I] Checking connection
2026-06-20 07:12:11,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 07:12:11,866 [I] connection success
2026-06-20 07:12:11,866 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 07:12:11,866 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 07:12:11,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 07:12:12,100 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 07:12:12,101 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:12:12,101 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 07:12:12,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.53.84:46643/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 07:12:12,348 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 07:12:12.799733 6357 mini_ranger_kms.cc:310] Created kms service
I20260620 07:12:12.984516 6357 mini_ranger_kms.cc:326] Created kudu user
I20260620 07:12:13.027781 6357 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 07:12:13 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939530, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
I20260620 07:12:13.401453 6357 mini_ranger_kms.cc:384] Added ranger policy
I20260620 07:12:13.401801 6357 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-kms:/tmp/dist-test-taskS8B4yM/build/asan/bin/postgresql.jar:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskS8B4yM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 07:12:13.401851 6357 mini_ranger_kms.cc:240] Using host: 127.6.53.84
I20260620 07:12:13.416637 6357 mini_ranger_kms.cc:276] Ranger KMS PID: 18610
I20260620 07:12:13.416775 6357 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.53.84:33309
Jun 20, 2026 7:12:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:12:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:12:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:12:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:12:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:12:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 07:12:15.702020 6357 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.253s user 0.069s sys 0.284s
I20260620 07:12:15.702314 6357 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 07:12:15.702509 6357 mini_ranger_kms.cc:401] 127.6.53.84:33309/kms/v1/keys
Jun 20 07:12:16 dist-test-slave-4n6h krb5kdc[18217](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939536, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.53.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 07:12:16 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939536, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.53.84@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
Jun 20 07:12:16 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939536, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.53.84@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
Jun 20 07:12:19 dist-test-slave-4n6h krb5kdc[18217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:12:19.074923 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:33363
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:33363
--ranger_config_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.53.84:33309/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:19.387320 18661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:19.387642 18661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:19.387795 18661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:19.395280 18661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:19.395470 18661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:19.395593 18661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:19.395689 18661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:19.395792 18661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:19.405941 18661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.53.84:33309/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:33363
--ranger_config_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:19.407368 18661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:19.408756 18661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:19.420990 18666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:19.421068 18667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:19.421088 18669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:19.421638 18661 server_base.cc:1034] running on GCE node
I20260620 07:12:19.422585 18661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:19.424109 18661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:19.425348 18661 hybrid_clock.cc:648] HybridClock initialized: now 1781939539425311 us; error 35 us; skew 500 ppm
I20260620 07:12:19.425716 18661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:19 dist-test-slave-4n6h krb5kdc[18217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939539, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:19.434171 18661 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:12:19.435792 18661 webserver.cc:460] Webserver started at http://127.6.53.126:46667/ using document root <none> and password file <none>
I20260620 07:12:19.436501 18661 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:19.436610 18661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:19.437004 18661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:12:19 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939539, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
I20260620 07:12:19.496194 18661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "88e6316a096b474689de8f2b01d9d4f9"
format_stamp: "Formatted at 2026-06-20 07:12:19 on dist-test-slave-4n6h"
server_key: "b0aae90dfcb62ecbd84bec27d6565bf6"
server_key_iv: "cede8f9aaed5d94b664e8dbe5fc47627"
server_key_version: "kuduclusterkey@0"
I20260620 07:12:19.497138 18661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "88e6316a096b474689de8f2b01d9d4f9"
format_stamp: "Formatted at 2026-06-20 07:12:19 on dist-test-slave-4n6h"
server_key: "b0aae90dfcb62ecbd84bec27d6565bf6"
server_key_iv: "cede8f9aaed5d94b664e8dbe5fc47627"
server_key_version: "kuduclusterkey@0"
I20260620 07:12:19.502997 18661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:12:19.520517 18676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:19.522110 18661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:12:19.522349 18661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "88e6316a096b474689de8f2b01d9d4f9"
format_stamp: "Formatted at 2026-06-20 07:12:19 on dist-test-slave-4n6h"
server_key: "b0aae90dfcb62ecbd84bec27d6565bf6"
server_key_iv: "cede8f9aaed5d94b664e8dbe5fc47627"
server_key_version: "kuduclusterkey@0"
I20260620 07:12:19.522598 18661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:19.534694 18661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:19.539374 18661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:19.539702 18661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:19.560120 18661 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:33363
I20260620 07:12:19.560138 18727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:33363 every 8 connection(s)
I20260620 07:12:19.561468 18661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:12:19.565136 18728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:19.569674 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 18661
I20260620 07:12:19.569888 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:12:19.570741 6357 external_mini_cluster.cc:1428] Setting key 9a80c327d69c04e1f261c60dfc7c71dc
I20260620 07:12:19.578426 18728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9: Bootstrap starting.
I20260620 07:12:19.582525 18728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:19.583616 18728 log.cc:824] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:19.586603 18728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9: No bootstrap required, opened a new log
Jun 20 07:12:19 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:12:19.595124 18728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e6316a096b474689de8f2b01d9d4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 33363 } }
I20260620 07:12:19.595525 18728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:19.595664 18728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e6316a096b474689de8f2b01d9d4f9, State: Initialized, Role: FOLLOWER
I20260620 07:12:19.596314 18728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e6316a096b474689de8f2b01d9d4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 33363 } }
I20260620 07:12:19.596578 18728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:12:19.596745 18728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:12:19.596958 18728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:19.598094 18731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:19.572869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.6.53.126:33363)
0620 07:12:19.578843 (+ 5974us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:19.578861 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:19.579221 (+ 360us) server_negotiation.cc:407] Connection header received
0620 07:12:19.581532 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:19.581555 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:19.581928 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:19.582343 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:19.583701 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:19.585023 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:19.586350 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:19.587112 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:19.587301 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:19.591054 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:19.591063 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:19.591075 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:19.591107 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:19.594167 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:19.594832 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:19.594838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:19.594844 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:19.594984 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:19.595343 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:19.595352 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:19.595354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:19.596048 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:19.596343 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:19.596778 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 07:12:19.597166 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5708,"thread_start_us":5494,"threads_started":1}
I20260620 07:12:19.600167 18728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e6316a096b474689de8f2b01d9d4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 33363 } }
I20260620 07:12:19.600688 18728 leader_election.cc:302] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e6316a096b474689de8f2b01d9d4f9; no voters:
I20260620 07:12:19.601389 18728 leader_election.cc:288] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:12:19.601495 18733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:19.602240 18733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 1 LEADER]: Becoming Leader. State: Replica: 88e6316a096b474689de8f2b01d9d4f9, State: Running, Role: LEADER
I20260620 07:12:19.602825 18733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e6316a096b474689de8f2b01d9d4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 33363 } }
I20260620 07:12:19.603629 18728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:19.605264 18735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e6316a096b474689de8f2b01d9d4f9. Latest consensus state: current_term: 1 leader_uuid: "88e6316a096b474689de8f2b01d9d4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e6316a096b474689de8f2b01d9d4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 33363 } } }
I20260620 07:12:19.605679 18735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:19.605973 18734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88e6316a096b474689de8f2b01d9d4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e6316a096b474689de8f2b01d9d4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 33363 } } }
I20260620 07:12:19.606333 18734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:19.608474 18738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:12:19.615829 18738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:12:19.616639 18728 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 07:12:19.635991 18738 catalog_manager.cc:1347] Generated new cluster ID: 192b9c54cb6049bbaeea58a69d1c0f95
I20260620 07:12:19.636184 18738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:12:19.663008 18738 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:12:19.664506 18738 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:12:19.688030 18738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9: Generated new TSK 0
I20260620 07:12:19.689237 18738 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:12:20.450107 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.53.84:33309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:33363
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:12:20 dist-test-slave-4n6h krb5kdc[18217](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939540, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 07:12:20.841677 18773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:20.842005 18773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:20.842167 18773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:20.850415 18773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:20.850561 18773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:20.850816 18773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:12:20.866703 18773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.53.84:33309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:33363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:20.868407 18773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:20.870148 18773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:20.887732 18784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:20.887800 18783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:20.894862 18786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:20.895852 18773 server_base.cc:1034] running on GCE node
I20260620 07:12:20.896643 18773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:20.897910 18773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:20.899219 18773 hybrid_clock.cc:648] HybridClock initialized: now 1781939540899179 us; error 68 us; skew 500 ppm
I20260620 07:12:20.899618 18773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:20 dist-test-slave-4n6h krb5kdc[18217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939540, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:20.903941 18773 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:12:20.907816 18773 webserver.cc:460] Webserver started at http://127.6.53.65:33443/ using document root <none> and password file <none>
I20260620 07:12:20.908766 18773 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:20.908911 18773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:20.909276 18773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:12:20 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939540, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
I20260620 07:12:20.933117 18773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "7ec3703d6b9c4d4c98275f64a15e9be0"
format_stamp: "Formatted at 2026-06-20 07:12:20 on dist-test-slave-4n6h"
server_key: "6bb7ee214f71f49d94259f11fe11ce68"
server_key_iv: "2e406648c58ca8a266ea4435eb5e8503"
server_key_version: "kuduclusterkey@0"
I20260620 07:12:20.934240 18773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "7ec3703d6b9c4d4c98275f64a15e9be0"
format_stamp: "Formatted at 2026-06-20 07:12:20 on dist-test-slave-4n6h"
server_key: "6bb7ee214f71f49d94259f11fe11ce68"
server_key_iv: "2e406648c58ca8a266ea4435eb5e8503"
server_key_version: "kuduclusterkey@0"
I20260620 07:12:20.945390 18773 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.006s sys 0.003s
I20260620 07:12:20.982980 18793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:20.987437 18773 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.001s sys 0.014s
I20260620 07:12:20.987696 18773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "7ec3703d6b9c4d4c98275f64a15e9be0"
format_stamp: "Formatted at 2026-06-20 07:12:20 on dist-test-slave-4n6h"
server_key: "6bb7ee214f71f49d94259f11fe11ce68"
server_key_iv: "2e406648c58ca8a266ea4435eb5e8503"
server_key_version: "kuduclusterkey@0"
I20260620 07:12:20.988044 18773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:21.029712 18773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:21.035089 18773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:21.035429 18773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:21.037103 18773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:21.038965 18773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:21.039072 18773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:21.039194 18773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:21.039255 18773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:21.103528 18773 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:42301
I20260620 07:12:21.103605 18905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:42301 every 8 connection(s)
I20260620 07:12:21.105436 18773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:12:21.113597 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 18773
I20260620 07:12:21.113803 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:12:21.114645 6357 external_mini_cluster.cc:1428] Setting key 419dc40b655bdeb7be0fb53bd43be442
Jun 20 07:12:21 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939540, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
I20260620 07:12:21.162839 18909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:21.109554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:38367 (local address 127.6.53.126:33363)
0620 07:12:21.136541 (+ 26987us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:21.136549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:21.136575 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:12:21.139456 (+ 2881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:21.139464 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:21.139577 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:21.139682 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:21.142569 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:21.143761 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:21.146192 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:21.146859 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:21.146978 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:21.151833 (+ 4855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:21.151840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:21.151845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:21.151887 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:21.157301 (+ 5414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:21.159275 (+ 1974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:21.159280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:21.159284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:21.159431 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:21.160634 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:21.160640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:21.160643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:21.161024 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:21.161174 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:21.162167 (+ 993us) server_negotiation.cc:299] Negotiation successful
0620 07:12:21.162426 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26759,"thread_start_us":26652,"threads_started":1}
I20260620 07:12:21.165694 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:21.108886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:33363 (local address 127.6.53.65:38367)
0620 07:12:21.135389 (+ 26503us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:21.135453 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:21.139004 (+ 3551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:21.140154 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:21.140169 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:21.140862 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:21.141714 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:21.141740 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:21.143966 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:21.143971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:21.145994 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:21.146008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:21.147327 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:21.147334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:21.147606 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:21.148463 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:21.148500 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:21.151557 (+ 3057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:21.158490 (+ 6933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:21.158500 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:21.158520 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:21.159111 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:21.160224 (+ 1113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:21.160229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:21.160233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:21.160479 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:21.161288 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:21.161296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:21.161744 (+ 448us) client_negotiation.cc:769] Sending connection context
0620 07:12:21.163550 (+ 1806us) client_negotiation.cc:240] Negotiation successful
0620 07:12:21.164081 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26175,"thread_start_us":25924,"threads_started":1}
I20260620 07:12:21.167878 18906 heartbeater.cc:347] Connected to a master server at 127.6.53.126:33363
I20260620 07:12:21.168217 18906 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:21.169014 18906 heartbeater.cc:511] Master 127.6.53.126:33363 requested a full tablet report, sending...
I20260620 07:12:21.171446 18693 ts_manager.cc:194] Registered new tserver with Master: 7ec3703d6b9c4d4c98275f64a15e9be0 (127.6.53.65:42301)
I20260620 07:12:21.173424 18693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:38367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:12:21.189289 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.53.84:33309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:33363
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:12:21 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939540, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
Jun 20 07:12:21 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939540, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
W20260620 07:12:21.601923 18912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:21.607717 18912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:21.607887 18912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:21.618925 18912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:21.619117 18912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:21.619314 18912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:12:21.634120 18912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.53.84:33309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:33363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:21.636104 18912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:21.637800 18912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:21.665182 18918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:21.666707 18919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:21.712525 18922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:21.714500 18912 server_base.cc:1034] running on GCE node
I20260620 07:12:21.715430 18912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:21.739542 18912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:21.743376 18912 hybrid_clock.cc:648] HybridClock initialized: now 1781939541743257 us; error 127 us; skew 500 ppm
I20260620 07:12:21.743870 18912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:21 dist-test-slave-4n6h krb5kdc[18217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939541, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:21.748199 18912 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:12:21.790638 18912 webserver.cc:460] Webserver started at http://127.6.53.66:43841/ using document root <none> and password file <none>
I20260620 07:12:21.791862 18912 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:21.792121 18912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:21.792587 18912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:12:21 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939541, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
I20260620 07:12:21.829468 18912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "49f64e1bb1a14ae4838e5aa47670bc1e"
format_stamp: "Formatted at 2026-06-20 07:12:21 on dist-test-slave-4n6h"
server_key: "e71310bc904b15218f784d728e6fdc41"
server_key_iv: "3f1c1268d8104a97cb7419f55fdfff25"
server_key_version: "kuduclusterkey@0"
I20260620 07:12:21.830619 18912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "49f64e1bb1a14ae4838e5aa47670bc1e"
format_stamp: "Formatted at 2026-06-20 07:12:21 on dist-test-slave-4n6h"
server_key: "e71310bc904b15218f784d728e6fdc41"
server_key_iv: "3f1c1268d8104a97cb7419f55fdfff25"
server_key_version: "kuduclusterkey@0"
I20260620 07:12:21.844717 18912 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.008s
I20260620 07:12:21.899868 18929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:21.903353 18912 fs_manager.cc:725] Time spent opening block manager: real 0.029s user 0.016s sys 0.012s
I20260620 07:12:21.903615 18912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "49f64e1bb1a14ae4838e5aa47670bc1e"
format_stamp: "Formatted at 2026-06-20 07:12:21 on dist-test-slave-4n6h"
server_key: "e71310bc904b15218f784d728e6fdc41"
server_key_iv: "3f1c1268d8104a97cb7419f55fdfff25"
server_key_version: "kuduclusterkey@0"
I20260620 07:12:21.903978 18912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:21.954783 18912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:21.960084 18912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:21.960455 18912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:21.974121 18912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:21.988297 18912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:21.988469 18912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 07:12:21.988642 18912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:21.988744 18912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:22.044368 18912 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:38577
I20260620 07:12:22.044410 19048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:38577 every 8 connection(s)
I20260620 07:12:22.046023 18912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:12:22.051687 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 18912
I20260620 07:12:22.051848 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:12:22.052450 6357 external_mini_cluster.cc:1428] Setting key cd393a96ba613f0ba5526758a445f66b
Jun 20 07:12:22 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939541, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
I20260620 07:12:22.087524 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:22.049491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:53645 (local address 127.6.53.126:33363)
0620 07:12:22.062703 (+ 13212us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:22.062711 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:22.066508 (+ 3797us) server_negotiation.cc:407] Connection header received
0620 07:12:22.069240 (+ 2732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:22.069245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:22.069351 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:22.069472 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:22.072095 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.072861 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.075181 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.075596 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.075742 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:22.080140 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:22.080145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:22.080148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:22.080181 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:22.083653 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:22.084624 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:22.084629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:22.084633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:22.084783 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:22.085319 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:22.085324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:22.085327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:22.085695 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:22.085837 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:22.086871 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0620 07:12:22.087132 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13075,"thread_start_us":12954,"threads_started":1}
I20260620 07:12:22.089805 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:22.050070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:33363 (local address 127.6.53.66:53645)
0620 07:12:22.066339 (+ 16269us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:22.066394 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:22.068859 (+ 2465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:22.070385 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:22.070397 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:22.071078 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:22.071925 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:22.071949 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.073028 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.073034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:22.075030 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:22.075044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.075947 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.075956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:22.076227 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:22.077072 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:22.077102 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:22.079887 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:22.083854 (+ 3967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:22.083864 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:22.083882 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:22.084477 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:22.084942 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:22.084946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:22.084950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:22.085188 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:22.085944 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:22.085953 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:22.086389 (+ 436us) client_negotiation.cc:769] Sending connection context
0620 07:12:22.087819 (+ 1430us) client_negotiation.cc:240] Negotiation successful
0620 07:12:22.088453 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15924,"thread_start_us":19125,"threads_started":1}
I20260620 07:12:22.091553 19049 heartbeater.cc:347] Connected to a master server at 127.6.53.126:33363
I20260620 07:12:22.091863 19049 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:22.092773 19049 heartbeater.cc:511] Master 127.6.53.126:33363 requested a full tablet report, sending...
I20260620 07:12:22.099354 18693 ts_manager.cc:194] Registered new tserver with Master: 49f64e1bb1a14ae4838e5aa47670bc1e (127.6.53.66:38577)
I20260620 07:12:22.100472 18693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:53645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:12:22.120062 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.53.84:33309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:33363
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:12:22.179641 18906 heartbeater.cc:503] Master 127.6.53.126:33363 was elected leader, sending a full tablet report...
W20260620 07:12:22.500248 19055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:22.500511 19055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:22.500661 19055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:22.508208 19055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:22.508306 19055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:22.508570 19055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:12:22.518621 19055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.53.84:33309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:33363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:22.519814 19055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:22.521055 19055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:22.533708 19060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:22.533807 19063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:22.533849 19061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:22.535281 19055 server_base.cc:1034] running on GCE node
I20260620 07:12:22.535815 19055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:22.536652 19055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:22.537846 19055 hybrid_clock.cc:648] HybridClock initialized: now 1781939542537811 us; error 34 us; skew 500 ppm
I20260620 07:12:22.538141 19055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:22 dist-test-slave-4n6h krb5kdc[18217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939542, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:22.541409 19055 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:12:22.543004 19055 webserver.cc:460] Webserver started at http://127.6.53.67:35485/ using document root <none> and password file <none>
I20260620 07:12:22.543756 19055 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:22.543919 19055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:22.544256 19055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:12:22 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939542, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
I20260620 07:12:22.560191 19055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "545913c80924416a92a58eabdbd45895"
format_stamp: "Formatted at 2026-06-20 07:12:22 on dist-test-slave-4n6h"
server_key: "3989f4087548d0b9e775575e2a373784"
server_key_iv: "b1ba16af8ffd5edefc0fdfd381ba2ca5"
server_key_version: "kuduclusterkey@0"
I20260620 07:12:22.561090 19055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "545913c80924416a92a58eabdbd45895"
format_stamp: "Formatted at 2026-06-20 07:12:22 on dist-test-slave-4n6h"
server_key: "3989f4087548d0b9e775575e2a373784"
server_key_iv: "b1ba16af8ffd5edefc0fdfd381ba2ca5"
server_key_version: "kuduclusterkey@0"
I20260620 07:12:22.566823 19055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 07:12:22.584525 19070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:22.585958 19055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:22.586151 19055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "545913c80924416a92a58eabdbd45895"
format_stamp: "Formatted at 2026-06-20 07:12:22 on dist-test-slave-4n6h"
server_key: "3989f4087548d0b9e775575e2a373784"
server_key_iv: "b1ba16af8ffd5edefc0fdfd381ba2ca5"
server_key_version: "kuduclusterkey@0"
I20260620 07:12:22.586444 19055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:22.610544 19055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:22.614490 19055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:22.614796 19055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:22.615729 19055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:22.616976 19055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:22.617051 19055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:22.617220 19055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:22.617269 19055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:22.648731 19055 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:35713
I20260620 07:12:22.648739 19182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:35713 every 8 connection(s)
I20260620 07:12:22.649927 19055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:12:22.660101 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 19055
I20260620 07:12:22.660274 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:12:22.660756 6357 external_mini_cluster.cc:1428] Setting key 13a3de225f62fa93cd5f7d74001d1dae
Jun 20 07:12:22 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939542, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:12:22.668465 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:22.652266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:38417 (local address 127.6.53.126:33363)
0620 07:12:22.652757 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:22.652763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:22.653579 (+ 816us) server_negotiation.cc:407] Connection header received
0620 07:12:22.655313 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:22.655317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:22.655382 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:22.655476 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:22.656720 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.657423 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.658967 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.659396 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.659490 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:22.663009 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:22.663015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:22.663019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:22.663079 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:22.665647 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:22.666325 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:22.666331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:22.666334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:22.666461 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:22.666885 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:22.666890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:22.666893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:22.667166 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:22.667314 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:22.667894 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 07:12:22.668105 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":228,"threads_started":1}
I20260620 07:12:22.669272 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:22.652753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:33363 (local address 127.6.53.67:38417)
0620 07:12:22.653426 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:22.653460 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:22.655075 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:22.655658 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:22.655665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:22.656082 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:22.656583 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:22.656599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.657597 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.657601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:22.658850 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:22.658860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.659579 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.659583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:22.659764 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:22.660447 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:22.660471 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:22.662816 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:22.665781 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:22.665789 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:22.665803 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:22.666183 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:22.666577 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:22.666583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:22.666586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:22.666770 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:22.667284 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:22.667292 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:22.667574 (+ 282us) client_negotiation.cc:769] Sending connection context
0620 07:12:22.667813 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 07:12:22.668275 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":270,"threads_started":1}
I20260620 07:12:22.670475 19183 heartbeater.cc:347] Connected to a master server at 127.6.53.126:33363
I20260620 07:12:22.670753 19183 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:22.671406 19183 heartbeater.cc:511] Master 127.6.53.126:33363 requested a full tablet report, sending...
I20260620 07:12:22.672673 18693 ts_manager.cc:194] Registered new tserver with Master: 545913c80924416a92a58eabdbd45895 (127.6.53.67:35713)
I20260620 07:12:22.673449 18693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:38417
I20260620 07:12:22.676052 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:12:22.677665 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 18773
Jun 20 07:12:22 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:12:22.695606 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:22.678577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47336 (local address 127.6.53.65:42301)
0620 07:12:22.679174 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:22.679183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:22.679214 (+ 31us) server_negotiation.cc:407] Connection header received
0620 07:12:22.679281 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:22.679287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:22.679558 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:22.679701 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:22.681658 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.682926 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.684413 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.684927 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.685027 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:22.689624 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:22.689638 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:22.689652 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:22.689686 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:22.692799 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:22.693590 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:22.693597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:22.693600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:22.693763 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:22.694153 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:22.694157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:22.694159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:22.694564 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:22.694796 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:22.695027 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 07:12:22.695210 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":257,"threads_started":1}
I20260620 07:12:22.696812 18901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:22.778901 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 18773
I20260620 07:12:22.795949 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 18912
Jun 20 07:12:22 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:12:22.812606 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:22.796855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.6.53.66:38577)
0620 07:12:22.797382 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:22.797389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:22.797422 (+ 33us) server_negotiation.cc:407] Connection header received
0620 07:12:22.797482 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:22.797488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:22.797696 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:22.797819 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:22.799080 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.800054 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.801316 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.801855 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.801966 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:22.806618 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:22.806631 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:22.806645 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:22.806689 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:22.809750 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:22.810593 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:22.810604 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:22.810608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:22.810773 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:22.811153 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:22.811156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:22.811158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:22.811514 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:22.811731 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:22.812021 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 07:12:22.812241 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":289,"threads_started":1}
I20260620 07:12:22.813827 19044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:22.895821 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 18912
I20260620 07:12:22.911522 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 19055
Jun 20 07:12:22 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:12:22.926723 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:22.912511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.6.53.67:35713)
0620 07:12:22.913115 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:22.913123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:22.913150 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:12:22.913213 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:22.913220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:22.913410 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:22.913559 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:22.914797 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.915761 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.917029 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.917562 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.917653 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:22.921620 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:22.921629 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:22.921642 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:22.921671 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:22.924337 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:22.924907 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:22.924913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:22.924916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:22.925028 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:22.925322 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:22.925325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:22.925327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:22.925644 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:22.925866 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:22.926131 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 07:12:22.926405 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":320,"threads_started":1}
I20260620 07:12:22.927933 19178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:23.011297 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 19055
I20260620 07:12:23.027330 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 18661
I20260620 07:12:23.028664 18723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:23.132470 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 18661
I20260620 07:12:23.157140 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:33363
--webserver_interface=127.6.53.126
--webserver_port=46667
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:33363
--ranger_config_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.53.84:33309/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.53.84:33309/kms with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:23.471549 19195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:23.471858 19195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:23.472016 19195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:23.479074 19195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:23.479214 19195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:23.479314 19195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:23.479413 19195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:23.479493 19195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:23.489315 19195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.53.84:33309/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:33363
--ranger_config_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=46667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:23.490607 19195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:23.491911 19195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:23.503273 19201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:23.503268 19203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:23.503328 19200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:23.504405 19195 server_base.cc:1034] running on GCE node
I20260620 07:12:23.505220 19195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:23.506784 19195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:23.508015 19195 hybrid_clock.cc:648] HybridClock initialized: now 1781939543507960 us; error 57 us; skew 500 ppm
I20260620 07:12:23.508407 19195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:23 dist-test-slave-4n6h krb5kdc[18217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939543, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:23.511773 19195 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:12:23.513350 19195 webserver.cc:460] Webserver started at http://127.6.53.126:46667/ using document root <none> and password file <none>
I20260620 07:12:23.513955 19195 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:23.514068 19195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:12:23 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939543, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
I20260620 07:12:23.571125 19195 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 07:12:23.574133 19211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:23.575587 19195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:12:23.575791 19195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "88e6316a096b474689de8f2b01d9d4f9"
format_stamp: "Formatted at 2026-06-20 07:12:19 on dist-test-slave-4n6h"
server_key: "b0aae90dfcb62ecbd84bec27d6565bf6"
server_key_iv: "cede8f9aaed5d94b664e8dbe5fc47627"
server_key_version: "kuduclusterkey@0"
I20260620 07:12:23.576750 19195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:23.594806 19195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:23.598794 19195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:23.599090 19195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:23.617970 19195 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:33363
I20260620 07:12:23.618005 19262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:33363 every 8 connection(s)
I20260620 07:12:23.619457 19195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:12:23.626492 19263 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:12:23.629464 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 19195
I20260620 07:12:23.630362 19263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9: Bootstrap starting.
I20260620 07:12:23.630614 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.53.84:33309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:42301
--local_ip_for_outbound_sockets=127.6.53.65
--tserver_master_addrs=127.6.53.126:33363
--webserver_port=33443
--webserver_interface=127.6.53.65
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.53.84:33309/kms with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:12:23.652179 19263 log.cc:824] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:23.661252 19263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:12:23.661769 19263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9: Bootstrap complete.
I20260620 07:12:23.670841 19263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e6316a096b474689de8f2b01d9d4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 33363 } }
I20260620 07:12:23.671694 19263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e6316a096b474689de8f2b01d9d4f9, State: Initialized, Role: FOLLOWER
I20260620 07:12:23.672291 19263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e6316a096b474689de8f2b01d9d4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 33363 } }
I20260620 07:12:23.672513 19263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:12:23.672627 19263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:12:23.672816 19263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:12:23.676591 19263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e6316a096b474689de8f2b01d9d4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 33363 } }
I20260620 07:12:23.677096 19263 leader_election.cc:302] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e6316a096b474689de8f2b01d9d4f9; no voters:
I20260620 07:12:23.677778 19263 leader_election.cc:288] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:12:23.677910 19267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:12:23.679240 19267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [term 2 LEADER]: Becoming Leader. State: Replica: 88e6316a096b474689de8f2b01d9d4f9, State: Running, Role: LEADER
I20260620 07:12:23.679996 19263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:23.679878 19267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e6316a096b474689de8f2b01d9d4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 33363 } }
I20260620 07:12:23.682859 19269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e6316a096b474689de8f2b01d9d4f9. Latest consensus state: current_term: 2 leader_uuid: "88e6316a096b474689de8f2b01d9d4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e6316a096b474689de8f2b01d9d4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 33363 } } }
I20260620 07:12:23.683216 19269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:23.683720 19268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "88e6316a096b474689de8f2b01d9d4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e6316a096b474689de8f2b01d9d4f9" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 33363 } } }
I20260620 07:12:23.683982 19268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:23.688311 19272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:12:23.695729 19272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:12:23.697625 19272 catalog_manager.cc:1259] Loaded cluster ID: 192b9c54cb6049bbaeea58a69d1c0f95
I20260620 07:12:23.697747 19272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:12:23.703011 19272 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:12:23.704931 19272 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9: Loaded TSK: 0
I20260620 07:12:23.706902 19272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:12:23.712504 19263 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 07:12:24.065891 19265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:24.066387 19265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:24.066584 19265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:24.078462 19265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:24.078608 19265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:24.078842 19265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:12:24.094655 19265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.53.84:33309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:42301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=33443
--tserver_master_addrs=127.6.53.126:33363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:24.096338 19265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:24.098106 19265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:24.142894 19302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:24.143843 19301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:24.148602 19305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:24.149755 19265 server_base.cc:1034] running on GCE node
I20260620 07:12:24.150800 19265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:24.153035 19265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:24.154392 19265 hybrid_clock.cc:648] HybridClock initialized: now 1781939544154318 us; error 55 us; skew 500 ppm
I20260620 07:12:24.154815 19265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:24 dist-test-slave-4n6h krb5kdc[18217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939544, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:24.159034 19265 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:12:24.191134 19265 webserver.cc:460] Webserver started at http://127.6.53.65:33443/ using document root <none> and password file <none>
I20260620 07:12:24.192258 19265 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:24.192507 19265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:12:24 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939544, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
I20260620 07:12:24.239199 19265 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.004s sys 0.001s
I20260620 07:12:24.248864 19313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:24.250770 19265 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.007s
I20260620 07:12:24.251021 19265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "7ec3703d6b9c4d4c98275f64a15e9be0"
format_stamp: "Formatted at 2026-06-20 07:12:20 on dist-test-slave-4n6h"
server_key: "6bb7ee214f71f49d94259f11fe11ce68"
server_key_iv: "2e406648c58ca8a266ea4435eb5e8503"
server_key_version: "kuduclusterkey@0"
I20260620 07:12:24.252542 19265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:24.309022 19265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:24.313571 19265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:24.313846 19265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:24.315045 19265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:24.316634 19265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:24.316752 19265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:24.316884 19265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:24.316975 19265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:24.395231 19265 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:42301
I20260620 07:12:24.395321 19425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:42301 every 8 connection(s)
I20260620 07:12:24.397015 19265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:12:24.403656 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 19265
I20260620 07:12:24.404649 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.53.84:33309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:38577
--local_ip_for_outbound_sockets=127.6.53.66
--tserver_master_addrs=127.6.53.126:33363
--webserver_port=43841
--webserver_interface=127.6.53.66
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.53.84:33309/kms with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:12:24 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939544, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:12:24.451030 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:24.400545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:33363 (local address 127.6.53.65:56219)
0620 07:12:24.412127 (+ 11582us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:24.412192 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:24.414857 (+ 2665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:24.418659 (+ 3802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:24.418671 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:24.419292 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:24.420024 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:24.420049 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:24.425829 (+ 5780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:24.425835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:24.427998 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:24.428032 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:24.429643 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:24.429652 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:24.429944 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:24.430778 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:24.430811 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:24.436456 (+ 5645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:24.445433 (+ 8977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:24.445444 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:24.445463 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:24.446171 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:24.446851 (+ 680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:24.446856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:24.446860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:24.447171 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:24.448237 (+ 1066us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:24.448246 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:24.448603 (+ 357us) client_negotiation.cc:769] Sending connection context
0620 07:12:24.448948 (+ 345us) client_negotiation.cc:240] Negotiation successful
0620 07:12:24.449584 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11232,"thread_start_us":14639,"threads_started":1}
I20260620 07:12:24.451030 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:24.400480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:56219 (local address 127.6.53.126:33363)
0620 07:12:24.416514 (+ 16034us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:24.416532 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:24.416579 (+ 47us) server_negotiation.cc:407] Connection header received
0620 07:12:24.417043 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:24.417069 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:24.417541 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:24.417924 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:24.423562 (+ 5638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:24.425604 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:24.428222 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:24.429338 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:24.429582 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:24.436847 (+ 7265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:24.436858 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:24.436880 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:24.436932 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:24.445219 (+ 8287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:24.446429 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:24.446440 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:24.446448 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:24.446708 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:24.447337 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:24.447343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:24.447346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:24.448058 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:24.448313 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:24.449420 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0620 07:12:24.449832 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15754,"thread_start_us":15541,"threads_started":1}
I20260620 07:12:24.455166 19426 heartbeater.cc:347] Connected to a master server at 127.6.53.126:33363
I20260620 07:12:24.455479 19426 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:24.456367 19426 heartbeater.cc:511] Master 127.6.53.126:33363 requested a full tablet report, sending...
W20260620 07:12:24.459709 19426 heartbeater.cc:644] Failed to heartbeat to 127.6.53.126:33363 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:12:24.459944 19426 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:24.460222 19426 heartbeater.cc:511] Master 127.6.53.126:33363 requested a full tablet report, sending...
I20260620 07:12:24.465519 19426 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:24.465848 19426 heartbeater.cc:511] Master 127.6.53.126:33363 requested a full tablet report, sending...
W20260620 07:12:24.467093 19426 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:12:24.854688 19430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:24.855175 19430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:24.855399 19430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:24.867041 19430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:24.867192 19430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:24.867408 19430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:12:24.883062 19430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.53.84:33309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:38577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=43841
--tserver_master_addrs=127.6.53.126:33363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:24.884774 19430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:24.886566 19430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:24.903231 19446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:24.904062 19447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:24.911412 19449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:24.912034 19430 server_base.cc:1034] running on GCE node
I20260620 07:12:24.912873 19430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:24.914183 19430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:24.915606 19430 hybrid_clock.cc:648] HybridClock initialized: now 1781939544915494 us; error 95 us; skew 500 ppm
I20260620 07:12:24.916021 19430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:24 dist-test-slave-4n6h krb5kdc[18217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939544, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:24.920279 19430 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:12:24.922178 19430 webserver.cc:460] Webserver started at http://127.6.53.66:43841/ using document root <none> and password file <none>
I20260620 07:12:24.923099 19430 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:24.923249 19430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:12:24 dist-test-slave-4n6h krb5kdc[18217](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939544, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 07:12:24 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939544, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
I20260620 07:12:24.976598 19430 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.004s
I20260620 07:12:24.980067 19457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:24.981738 19430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:12:24.981894 19430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "49f64e1bb1a14ae4838e5aa47670bc1e"
format_stamp: "Formatted at 2026-06-20 07:12:21 on dist-test-slave-4n6h"
server_key: "e71310bc904b15218f784d728e6fdc41"
server_key_iv: "3f1c1268d8104a97cb7419f55fdfff25"
server_key_version: "kuduclusterkey@0"
I20260620 07:12:24.983234 19430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:25.015275 19430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:25.020649 19430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:25.020946 19430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:25.022174 19430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:25.023890 19430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:25.024015 19430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:25.024137 19430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:25.024219 19430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:25.071534 19430 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:38577
I20260620 07:12:25.071556 19570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:38577 every 8 connection(s)
I20260620 07:12:25.073347 19430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:12:25.077272 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 19430
I20260620 07:12:25.078085 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.53.84:33309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:35713
--local_ip_for_outbound_sockets=127.6.53.67
--tserver_master_addrs=127.6.53.126:33363
--webserver_port=35485
--webserver_interface=127.6.53.67
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.53.84:33309/kms with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:12:25 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939544, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:12:25.101265 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:25.076021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:45171 (local address 127.6.53.126:33363)
0620 07:12:25.076536 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:25.076542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:25.079638 (+ 3096us) server_negotiation.cc:407] Connection header received
0620 07:12:25.079773 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:25.079779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:25.079867 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:25.079976 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:25.082314 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.083472 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.085833 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.086479 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.086588 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:25.092973 (+ 6385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:25.092981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:25.092986 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:25.093026 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:25.097438 (+ 4412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:25.098443 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:25.098449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:25.098452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:25.098595 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:25.099153 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:25.099159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:25.099162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:25.099539 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:25.099676 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:25.100619 (+ 943us) server_negotiation.cc:299] Negotiation successful
0620 07:12:25.100839 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":241,"threads_started":1}
I20260620 07:12:25.103740 19574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:25.076497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:33363 (local address 127.6.53.66:45171)
0620 07:12:25.077256 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:25.077298 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:25.079270 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:25.080530 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:25.080542 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:25.081242 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:25.082095 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:25.082119 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.083695 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.083700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:25.085645 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:25.085659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.086960 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.086967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:25.087242 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:25.088474 (+ 1232us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:25.088509 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:25.092696 (+ 4187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:25.097616 (+ 4920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:25.097626 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:25.097645 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:25.098281 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:25.098739 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:25.098744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:25.098747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:25.099011 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:25.099758 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:25.099766 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:25.100172 (+ 406us) client_negotiation.cc:769] Sending connection context
0620 07:12:25.101760 (+ 1588us) client_negotiation.cc:240] Negotiation successful
0620 07:12:25.102354 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":328,"threads_started":1}
I20260620 07:12:25.105631 19571 heartbeater.cc:347] Connected to a master server at 127.6.53.126:33363
I20260620 07:12:25.105962 19571 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:25.106927 19571 heartbeater.cc:511] Master 127.6.53.126:33363 requested a full tablet report, sending...
I20260620 07:12:25.109052 19228 ts_manager.cc:194] Registered new tserver with Master: 49f64e1bb1a14ae4838e5aa47670bc1e (127.6.53.66:38577)
I20260620 07:12:25.111315 19228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:45171
Jun 20 07:12:25 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939544, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
I20260620 07:12:25.467217 19426 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:25.467653 19426 heartbeater.cc:511] Master 127.6.53.126:33363 requested a full tablet report, sending...
I20260620 07:12:25.468933 19228 ts_manager.cc:194] Registered new tserver with Master: 7ec3703d6b9c4d4c98275f64a15e9be0 (127.6.53.65:42301)
I20260620 07:12:25.470175 19228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:56219
W20260620 07:12:25.525563 19575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:25.525915 19575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:25.526116 19575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:25.537374 19575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:25.537514 19575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:25.537744 19575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:12:25.553077 19575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.53.84:33309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:35713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=35485
--tserver_master_addrs=127.6.53.126:33363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:25.554702 19575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:25.556263 19575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:25.576953 19580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:25.580314 19581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:25.580477 19583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:25.581948 19575 server_base.cc:1034] running on GCE node
I20260620 07:12:25.582741 19575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:25.583932 19575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:25.585233 19575 hybrid_clock.cc:648] HybridClock initialized: now 1781939545585172 us; error 49 us; skew 500 ppm
I20260620 07:12:25.585602 19575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:25 dist-test-slave-4n6h krb5kdc[18217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939545, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:25.589859 19575 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:12:25.593331 19575 webserver.cc:460] Webserver started at http://127.6.53.67:35485/ using document root <none> and password file <none>
I20260620 07:12:25.594237 19575 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:25.594412 19575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:12:25 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939545, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
I20260620 07:12:25.623319 19575 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.004s sys 0.000s
I20260620 07:12:25.627043 19591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:25.628858 19575 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:12:25.629035 19575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "545913c80924416a92a58eabdbd45895"
format_stamp: "Formatted at 2026-06-20 07:12:22 on dist-test-slave-4n6h"
server_key: "3989f4087548d0b9e775575e2a373784"
server_key_iv: "b1ba16af8ffd5edefc0fdfd381ba2ca5"
server_key_version: "kuduclusterkey@0"
I20260620 07:12:25.630350 19575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 07:12:25 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939544, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for HTTP/127.6.53.84@KRBTEST.COM
I20260620 07:12:25.681034 19575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:25.686487 19575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:25.686858 19575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:25.691161 19575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:25.699393 19575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:25.699524 19575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:25.699647 19575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:25.699726 19575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:25.888229 19575 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:35713
I20260620 07:12:25.888285 19711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:35713 every 8 connection(s)
I20260620 07:12:25.889612 19575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:12:25.891559 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 19575
Jun 20 07:12:25 dist-test-slave-4n6h krb5kdc[18217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939545, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:12:25.915773 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:25.893122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.6.53.126:33363)
0620 07:12:25.898111 (+ 4989us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:25.898118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:25.898142 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:12:25.898233 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:25.898240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:25.898382 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:25.900489 (+ 2107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:25.900668 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.901883 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.903987 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.904621 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.904748 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:25.908032 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:25.908054 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:25.908059 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:25.908101 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:25.911657 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:25.912502 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:25.912507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:25.912510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:25.912714 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:25.914125 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:25.914131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:25.914135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:25.914538 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:25.914801 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:25.915159 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 07:12:25.915386 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4852,"spinlock_wait_cycles":3968,"thread_start_us":4683,"threads_started":1}
I20260620 07:12:25.918362 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:25.892765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:33363 (local address 127.6.53.67:49061)
0620 07:12:25.898825 (+ 6060us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:25.898878 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:25.900602 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:25.901322 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:25.901333 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:25.901751 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:25.902516 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:25.902540 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.904169 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.904173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:25.905836 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:25.905852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.906912 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.906920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:25.907245 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:25.908002 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:25.908030 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:25.910392 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:25.913764 (+ 3372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:25.913773 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:25.913790 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:25.914353 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:25.914751 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:25.914756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:25.914759 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:25.915028 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:25.915716 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:25.915723 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:25.916132 (+ 409us) client_negotiation.cc:769] Sending connection context
0620 07:12:25.916461 (+ 329us) client_negotiation.cc:240] Negotiation successful
0620 07:12:25.916998 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5789,"thread_start_us":5586,"threads_started":1}
I20260620 07:12:25.918521 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:25.892151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:49061 (local address 127.6.53.126:33363)
0620 07:12:25.892631 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:25.892638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:25.898988 (+ 6350us) server_negotiation.cc:407] Connection header received
0620 07:12:25.900871 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:25.900876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:25.900962 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:25.901075 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:25.902763 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.903988 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.905994 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.906666 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.906784 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:25.910650 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:25.910655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:25.910658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:25.910709 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:25.913632 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:25.914465 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:25.914472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:25.914475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:25.914617 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:25.915158 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:25.915162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:25.915165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:25.915500 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:25.915624 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:25.916697 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0620 07:12:25.916893 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":773,"threads_started":1}
I20260620 07:12:25.919932 19712 heartbeater.cc:347] Connected to a master server at 127.6.53.126:33363
I20260620 07:12:25.920190 19712 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:25.920796 19712 heartbeater.cc:511] Master 127.6.53.126:33363 requested a full tablet report, sending...
I20260620 07:12:25.922096 19228 ts_manager.cc:194] Registered new tserver with Master: 545913c80924416a92a58eabdbd45895 (127.6.53.67:35713)
I20260620 07:12:25.922914 19228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:49061
I20260620 07:12:25.931106 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:12:25.947649 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:25.934628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.6.53.126:33363)
0620 07:12:25.934797 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:25.934802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:25.935465 (+ 663us) server_negotiation.cc:407] Connection header received
0620 07:12:25.935670 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:25.935676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:25.935776 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:25.935868 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:25.937307 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.937994 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.939377 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.939798 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.939931 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:25.942466 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:25.942471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:25.942474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:25.942503 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:25.945122 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:25.945762 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:25.945767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:25.945770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:25.945890 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:25.946300 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:25.946305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:25.946308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:25.946586 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:25.946823 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:25.947243 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 07:12:25.947423 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:12:25.953300 19228 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:12:25.960824 19228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:12:25.996311 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:25.975953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:38577 (local address 127.0.0.1:55140)
0620 07:12:25.984812 (+ 8859us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:25.984868 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:25.985087 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:25.986030 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:25.986042 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:25.986100 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:25.986779 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:25.986793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.989340 (+ 2547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.989349 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:25.992466 (+ 3117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:25.992485 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.995139 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.995147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:25.995396 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:25.995461 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 07:12:25.995611 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 07:12:25.995827 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8556,"thread_start_us":10586,"threads_started":1}
I20260620 07:12:25.996351 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:25.976115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.6.53.66:38577)
0620 07:12:25.983680 (+ 7565us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:25.983692 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:25.985314 (+ 1622us) server_negotiation.cc:407] Connection header received
0620 07:12:25.985450 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:25.985459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:25.985732 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:25.985991 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:25.987011 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.989037 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.992688 (+ 3651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.994539 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.994659 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:25.994895 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:25.995700 (+ 805us) server_negotiation.cc:299] Negotiation successful
0620 07:12:25.995909 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7398,"thread_start_us":7286,"threads_started":1}
I20260620 07:12:25.997812 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:25.976976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.6.53.67:35713)
0620 07:12:25.984252 (+ 7276us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:25.984263 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:25.987231 (+ 2968us) server_negotiation.cc:407] Connection header received
0620 07:12:25.987364 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:25.987373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:25.987589 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:25.987799 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:25.989106 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.991098 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.994364 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.996015 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.996155 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:25.997108 (+ 953us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:25.997297 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 07:12:25.997448 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7082,"thread_start_us":6913,"threads_started":1}
I20260620 07:12:25.997876 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:25.976889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:35713 (local address 127.0.0.1:36038)
0620 07:12:25.986913 (+ 10024us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:25.986934 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:25.987077 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:25.987916 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:25.987922 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:25.987954 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:25.988471 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:25.988488 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.991358 (+ 2870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.991365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:25.994097 (+ 2732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:25.994115 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.996805 (+ 2690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.996813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:25.997041 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:25.997094 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 07:12:25.997240 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 07:12:25.997429 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9863,"thread_start_us":7442,"threads_started":1}
I20260620 07:12:26.002143 19640 tablet_service.cc:1467] Processing CreateTablet for tablet 7c154e3080b74138b7082e8e87317f7b (DEFAULT_TABLE table=test-table [id=d6717e21f43a4fcb9e756ec19f6f1c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:26.002143 19506 tablet_service.cc:1467] Processing CreateTablet for tablet 7c154e3080b74138b7082e8e87317f7b (DEFAULT_TABLE table=test-table [id=d6717e21f43a4fcb9e756ec19f6f1c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:26.003571 19506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c154e3080b74138b7082e8e87317f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:26.003569 19640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c154e3080b74138b7082e8e87317f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:26.005076 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:25.993187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42301 (local address 127.0.0.1:47344)
0620 07:12:25.996773 (+ 3586us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:25.996807 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:25.996916 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:25.998719 (+ 1803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:25.998723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:25.998751 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:25.998928 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:25.998938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.000574 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.000602 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:26.002477 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:26.002495 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.004152 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.004159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:26.004386 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:26.004437 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 07:12:26.004577 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 07:12:26.004732 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3464,"thread_start_us":4097,"threads_started":1}
I20260620 07:12:26.005211 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:25.987329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.6.53.65:42301)
0620 07:12:25.997956 (+ 10627us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:25.997969 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:25.998005 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:12:25.998125 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:25.998138 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:25.998440 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:25.998723 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:25.999105 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.000417 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.002668 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.003968 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.004141 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:26.004392 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:26.004620 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 07:12:26.004801 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10425,"thread_start_us":10275,"threads_started":1}
I20260620 07:12:26.009255 19361 tablet_service.cc:1467] Processing CreateTablet for tablet 7c154e3080b74138b7082e8e87317f7b (DEFAULT_TABLE table=test-table [id=d6717e21f43a4fcb9e756ec19f6f1c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:26.010953 19361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c154e3080b74138b7082e8e87317f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:26.019078 19733 tablet_bootstrap.cc:492] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e: Bootstrap starting.
I20260620 07:12:26.022943 19734 tablet_bootstrap.cc:492] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895: Bootstrap starting.
I20260620 07:12:26.023351 19733 tablet_bootstrap.cc:654] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:26.024843 19733 log.cc:824] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:26.026312 19734 tablet_bootstrap.cc:654] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:26.027444 19734 log.cc:824] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:26.028223 19735 tablet_bootstrap.cc:492] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0: Bootstrap starting.
I20260620 07:12:26.028389 19733 tablet_bootstrap.cc:492] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e: No bootstrap required, opened a new log
I20260620 07:12:26.028694 19733 ts_tablet_manager.cc:1397] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 07:12:26.030598 19734 tablet_bootstrap.cc:492] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895: No bootstrap required, opened a new log
I20260620 07:12:26.030854 19734 ts_tablet_manager.cc:1397] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 07:12:26.031488 19735 tablet_bootstrap.cc:654] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:26.032622 19735 log.cc:824] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:26.035940 19735 tablet_bootstrap.cc:492] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0: No bootstrap required, opened a new log
I20260620 07:12:26.036193 19735 ts_tablet_manager.cc:1397] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 07:12:26.038097 19733 raft_consensus.cc:348] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f64e1bb1a14ae4838e5aa47670bc1e" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38577 } } peers { permanent_uuid: "545913c80924416a92a58eabdbd45895" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35713 } } peers { permanent_uuid: "7ec3703d6b9c4d4c98275f64a15e9be0" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42301 } }
I20260620 07:12:26.038579 19733 raft_consensus.cc:374] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:26.038712 19733 raft_consensus.cc:729] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f64e1bb1a14ae4838e5aa47670bc1e, State: Initialized, Role: FOLLOWER
I20260620 07:12:26.039392 19733 consensus_queue.cc:260] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f64e1bb1a14ae4838e5aa47670bc1e" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38577 } } peers { permanent_uuid: "545913c80924416a92a58eabdbd45895" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35713 } } peers { permanent_uuid: "7ec3703d6b9c4d4c98275f64a15e9be0" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42301 } }
I20260620 07:12:26.040624 19571 heartbeater.cc:503] Master 127.6.53.126:33363 was elected leader, sending a full tablet report...
I20260620 07:12:26.041361 19733 ts_tablet_manager.cc:1428] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260620 07:12:26.042169 19734 raft_consensus.cc:348] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f64e1bb1a14ae4838e5aa47670bc1e" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38577 } } peers { permanent_uuid: "545913c80924416a92a58eabdbd45895" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35713 } } peers { permanent_uuid: "7ec3703d6b9c4d4c98275f64a15e9be0" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42301 } }
I20260620 07:12:26.042585 19734 raft_consensus.cc:374] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:26.042685 19734 raft_consensus.cc:729] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545913c80924416a92a58eabdbd45895, State: Initialized, Role: FOLLOWER
I20260620 07:12:26.043231 19734 consensus_queue.cc:260] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f64e1bb1a14ae4838e5aa47670bc1e" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38577 } } peers { permanent_uuid: "545913c80924416a92a58eabdbd45895" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35713 } } peers { permanent_uuid: "7ec3703d6b9c4d4c98275f64a15e9be0" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42301 } }
I20260620 07:12:26.045903 19712 heartbeater.cc:503] Master 127.6.53.126:33363 was elected leader, sending a full tablet report...
I20260620 07:12:26.046521 19734 ts_tablet_manager.cc:1428] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260620 07:12:26.047340 19735 raft_consensus.cc:348] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f64e1bb1a14ae4838e5aa47670bc1e" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38577 } } peers { permanent_uuid: "545913c80924416a92a58eabdbd45895" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35713 } } peers { permanent_uuid: "7ec3703d6b9c4d4c98275f64a15e9be0" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42301 } }
I20260620 07:12:26.047784 19735 raft_consensus.cc:374] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:26.047891 19735 raft_consensus.cc:729] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec3703d6b9c4d4c98275f64a15e9be0, State: Initialized, Role: FOLLOWER
I20260620 07:12:26.048405 19735 consensus_queue.cc:260] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f64e1bb1a14ae4838e5aa47670bc1e" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38577 } } peers { permanent_uuid: "545913c80924416a92a58eabdbd45895" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35713 } } peers { permanent_uuid: "7ec3703d6b9c4d4c98275f64a15e9be0" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42301 } }
I20260620 07:12:26.049510 19426 heartbeater.cc:503] Master 127.6.53.126:33363 was elected leader, sending a full tablet report...
I20260620 07:12:26.049968 19735 ts_tablet_manager.cc:1428] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260620 07:12:26.075541 19572 tablet.cc:2369] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:12:26.101908 19739 raft_consensus.cc:482] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:26.102188 19739 raft_consensus.cc:504] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f64e1bb1a14ae4838e5aa47670bc1e" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38577 } } peers { permanent_uuid: "545913c80924416a92a58eabdbd45895" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35713 } } peers { permanent_uuid: "7ec3703d6b9c4d4c98275f64a15e9be0" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42301 } }
I20260620 07:12:26.104004 19739 leader_election.cc:288] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 545913c80924416a92a58eabdbd45895 (127.6.53.67:35713), 7ec3703d6b9c4d4c98275f64a15e9be0 (127.6.53.65:42301)
I20260620 07:12:26.110484 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.104392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:36145 (local address 127.6.53.67:35713)
0620 07:12:26.104669 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:26.104679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:26.105401 (+ 722us) server_negotiation.cc:407] Connection header received
0620 07:12:26.105647 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:26.105651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:26.105710 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:26.105802 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:26.106287 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.107002 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.108725 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.109558 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.109704 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:26.109805 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:26.110077 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 07:12:26.110202 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 07:12:26.110642 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.104222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:35713 (local address 127.6.53.66:36145)
0620 07:12:26.105290 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:26.105306 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:26.105497 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:26.105817 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:26.105821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:26.105843 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:26.106137 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:26.106146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.107135 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.107142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:26.108572 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:26.108583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.109754 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.109760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:26.109924 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:26.109947 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:12:26.110052 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 07:12:26.110230 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":1,"thread_start_us":664,"threads_started":1}
I20260620 07:12:26.110899 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.104513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:49107 (local address 127.6.53.65:42301)
0620 07:12:26.104757 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:26.104765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:26.105486 (+ 721us) server_negotiation.cc:407] Connection header received
0620 07:12:26.105686 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:26.105690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:26.105748 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:26.105860 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:26.106698 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.107379 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.109095 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.109905 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.110050 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:26.110157 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:26.110551 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 07:12:26.110662 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:12:26.111199 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.104265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42301 (local address 127.6.53.66:49107)
0620 07:12:26.105380 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:26.105394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:26.105534 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:26.105874 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:26.105878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:26.105904 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:26.106539 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:26.106548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.107510 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.107515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:26.108906 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:26.108917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.110192 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.110203 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:26.110396 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:26.110418 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:12:26.110519 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 07:12:26.110815 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"mutex_wait_us":20,"thread_start_us":794,"threads_started":1}
I20260620 07:12:26.111717 19660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c154e3080b74138b7082e8e87317f7b" candidate_uuid: "49f64e1bb1a14ae4838e5aa47670bc1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "545913c80924416a92a58eabdbd45895" is_pre_election: true
I20260620 07:12:26.112198 19660 raft_consensus.cc:2454] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49f64e1bb1a14ae4838e5aa47670bc1e in term 0.
I20260620 07:12:26.112154 19381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c154e3080b74138b7082e8e87317f7b" candidate_uuid: "49f64e1bb1a14ae4838e5aa47670bc1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec3703d6b9c4d4c98275f64a15e9be0" is_pre_election: true
I20260620 07:12:26.112596 19381 raft_consensus.cc:2454] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49f64e1bb1a14ae4838e5aa47670bc1e in term 0.
I20260620 07:12:26.112823 19458 leader_election.cc:302] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f64e1bb1a14ae4838e5aa47670bc1e, 545913c80924416a92a58eabdbd45895; no voters:
I20260620 07:12:26.113221 19739 raft_consensus.cc:2781] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:12:26.113369 19739 raft_consensus.cc:482] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:12:26.113487 19739 raft_consensus.cc:3037] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:26.115696 19739 raft_consensus.cc:504] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f64e1bb1a14ae4838e5aa47670bc1e" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38577 } } peers { permanent_uuid: "545913c80924416a92a58eabdbd45895" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35713 } } peers { permanent_uuid: "7ec3703d6b9c4d4c98275f64a15e9be0" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42301 } }
I20260620 07:12:26.116304 19739 leader_election.cc:288] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [CANDIDATE]: Term 1 election: Requested vote from peers 545913c80924416a92a58eabdbd45895 (127.6.53.67:35713), 7ec3703d6b9c4d4c98275f64a15e9be0 (127.6.53.65:42301)
I20260620 07:12:26.116698 19381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c154e3080b74138b7082e8e87317f7b" candidate_uuid: "49f64e1bb1a14ae4838e5aa47670bc1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec3703d6b9c4d4c98275f64a15e9be0"
I20260620 07:12:26.116698 19660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c154e3080b74138b7082e8e87317f7b" candidate_uuid: "49f64e1bb1a14ae4838e5aa47670bc1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "545913c80924416a92a58eabdbd45895"
I20260620 07:12:26.116907 19660 raft_consensus.cc:3037] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:26.116907 19381 raft_consensus.cc:3037] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:26.120083 19381 raft_consensus.cc:2454] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49f64e1bb1a14ae4838e5aa47670bc1e in term 1.
I20260620 07:12:26.120083 19660 raft_consensus.cc:2454] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49f64e1bb1a14ae4838e5aa47670bc1e in term 1.
I20260620 07:12:26.120504 19461 leader_election.cc:302] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f64e1bb1a14ae4838e5aa47670bc1e, 7ec3703d6b9c4d4c98275f64a15e9be0; no voters:
I20260620 07:12:26.120877 19739 raft_consensus.cc:2781] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:26.121289 19739 raft_consensus.cc:686] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [term 1 LEADER]: Becoming Leader. State: Replica: 49f64e1bb1a14ae4838e5aa47670bc1e, State: Running, Role: LEADER
I20260620 07:12:26.121704 19739 consensus_queue.cc:237] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f64e1bb1a14ae4838e5aa47670bc1e" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38577 } } peers { permanent_uuid: "545913c80924416a92a58eabdbd45895" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35713 } } peers { permanent_uuid: "7ec3703d6b9c4d4c98275f64a15e9be0" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42301 } }
I20260620 07:12:26.126045 19228 catalog_manager.cc:5495] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 49f64e1bb1a14ae4838e5aa47670bc1e (127.6.53.66). New cstate: current_term: 1 leader_uuid: "49f64e1bb1a14ae4838e5aa47670bc1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f64e1bb1a14ae4838e5aa47670bc1e" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 38577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "545913c80924416a92a58eabdbd45895" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec3703d6b9c4d4c98275f64a15e9be0" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42301 } health_report { overall_health: UNKNOWN } } }
W20260620 07:12:26.141281 19713 tablet.cc:2369] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:12:26.151146 19427 tablet.cc:2369] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:12:26.160890 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.155628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.6.53.66:38577)
0620 07:12:26.155834 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:26.155840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:26.155857 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:12:26.155957 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:26.155961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:26.156033 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:26.156177 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:12:26.156577 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.157278 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.158796 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.159187 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.159279 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:26.159704 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:12:26.160174 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:12:26.160283 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:26.160426 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 07:12:26.160621 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:12:26.168594 19381 raft_consensus.cc:1261] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0 [term 1 FOLLOWER]: Refusing update from remote peer 49f64e1bb1a14ae4838e5aa47670bc1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:26.168678 19660 raft_consensus.cc:1261] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895 [term 1 FOLLOWER]: Refusing update from remote peer 49f64e1bb1a14ae4838e5aa47670bc1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:26.169574 19744 consensus_queue.cc:1035] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "545913c80924416a92a58eabdbd45895" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:26.169893 19739 consensus_queue.cc:1035] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec3703d6b9c4d4c98275f64a15e9be0" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:26.182159 19748 mvcc.cc:205] Tried to move back new op lower bound from 7298824381098156032 to 7298824380918673408. Current Snapshot: MvccSnapshot[applied={T|T < 7298824381098156032}]
I20260620 07:12:26.197136 19227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:26.197496 19227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:26.201187 19227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88e6316a096b474689de8f2b01d9d4f9: Sending DeleteTablet for 3 replicas of tablet 7c154e3080b74138b7082e8e87317f7b
I20260620 07:12:26.202376 19640 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c154e3080b74138b7082e8e87317f7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:26 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:36038
I20260620 07:12:26.202625 19361 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c154e3080b74138b7082e8e87317f7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:26 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:47344
I20260620 07:12:26.202734 19506 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c154e3080b74138b7082e8e87317f7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:26 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:55140
I20260620 07:12:26.212107 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 19265
I20260620 07:12:26.221042 19757 tablet_replica.cc:330] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895: stopping tablet replica
I20260620 07:12:26.222169 19757 raft_consensus.cc:2229] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:26.222864 19759 tablet_replica.cc:330] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e: stopping tablet replica
I20260620 07:12:26.222887 19757 raft_consensus.cc:2258] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:26.223287 19759 raft_consensus.cc:2229] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:12:26.223994 19759 raft_consensus.cc:2258] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:26.225339 19757 ts_tablet_manager.cc:1905] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:26.225533 19758 tablet_replica.cc:330] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0: stopping tablet replica
I20260620 07:12:26.226045 19759 ts_tablet_manager.cc:1905] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:26.226353 19758 raft_consensus.cc:2229] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:26.227213 19758 raft_consensus.cc:2258] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:26.232092 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.213509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.6.53.65:42301)
0620 07:12:26.214122 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:26.214129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:26.214152 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:12:26.214213 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:26.214219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:26.214377 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:26.214502 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:26.216766 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.218352 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.220310 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.221226 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.221413 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:26.224848 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:26.224863 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:26.224880 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:26.224925 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:26.228979 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:26.229706 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:26.229714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:26.229718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:26.229896 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:26.230354 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:26.230360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:26.230363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:26.230791 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:26.231067 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:26.231366 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 07:12:26.231578 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482}
I20260620 07:12:26.232424 19757 ts_tablet_manager.cc:1918] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:26.232620 19757 log.cc:1196] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/7c154e3080b74138b7082e8e87317f7b
I20260620 07:12:26.233079 19421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:26.233517 19759 ts_tablet_manager.cc:1918] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:26.233709 19759 log.cc:1196] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/7c154e3080b74138b7082e8e87317f7b
I20260620 07:12:26.233865 19757 ts_tablet_manager.cc:1939] T 7c154e3080b74138b7082e8e87317f7b P 545913c80924416a92a58eabdbd45895: Deleting consensus metadata
I20260620 07:12:26.234342 19759 ts_tablet_manager.cc:1939] T 7c154e3080b74138b7082e8e87317f7b P 49f64e1bb1a14ae4838e5aa47670bc1e: Deleting consensus metadata
I20260620 07:12:26.236136 19212 catalog_manager.cc:4841] TS 545913c80924416a92a58eabdbd45895 (127.6.53.67:35713): tablet 7c154e3080b74138b7082e8e87317f7b (table test-table [id=d6717e21f43a4fcb9e756ec19f6f1c47]) successfully deleted
I20260620 07:12:26.236132 19215 catalog_manager.cc:4841] TS 49f64e1bb1a14ae4838e5aa47670bc1e (127.6.53.66:38577): tablet 7c154e3080b74138b7082e8e87317f7b (table test-table [id=d6717e21f43a4fcb9e756ec19f6f1c47]) successfully deleted
I20260620 07:12:26.233266 19758 ts_tablet_manager.cc:1905] T 7c154e3080b74138b7082e8e87317f7b P 7ec3703d6b9c4d4c98275f64a15e9be0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:26.325253 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 19265
W20260620 07:12:26.348372 19215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.53.65:42301 (error 108)
I20260620 07:12:26.348634 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 19430
I20260620 07:12:26.350737 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.350300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42301 (local address 127.0.0.1:47364)
0620 07:12:26.350496 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:26.350587 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 07:12:26.351056 19215 catalog_manager.cc:4584] TS 7ec3703d6b9c4d4c98275f64a15e9be0 (127.6.53.65:42301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c154e3080b74138b7082e8e87317f7b: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42301: connect: Connection refused (error 111)
I20260620 07:12:26.364706 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.349410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.6.53.66:38577)
0620 07:12:26.349882 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:26.349889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:26.349909 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:12:26.349976 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:26.349984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:26.350065 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:26.350176 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:26.351858 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.352702 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.354577 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.354997 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.355092 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:26.358388 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:26.358402 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:26.358416 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:26.358510 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:26.361526 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:26.362703 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:26.362710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:26.362713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:26.362888 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:26.363280 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:26.363283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:26.363285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:26.363649 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:26.363911 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:26.364212 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 07:12:26.364389 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307}
I20260620 07:12:26.365644 19566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:26.404850 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.404232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42301 (local address 127.0.0.1:47378)
0620 07:12:26.404495 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:26.404628 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 07:12:26.406003 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.405552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42301 (local address 127.0.0.1:47384)
0620 07:12:26.405714 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:26.405815 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 07:12:26.466523 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.465961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42301 (local address 127.0.0.1:47386)
0620 07:12:26.466207 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:26.466344 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 07:12:26.467592 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.467206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42301 (local address 127.0.0.1:47390)
0620 07:12:26.467365 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:26.467450 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 07:12:26.478551 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 19430
I20260620 07:12:26.497707 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 19575
I20260620 07:12:26.511752 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.498534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.6.53.67:35713)
0620 07:12:26.498781 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:26.498786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:26.498803 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:12:26.498969 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:26.498973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:26.499035 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:26.499123 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:26.500463 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.501174 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.502734 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.503104 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.503192 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:26.505928 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:26.505940 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:26.505956 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:26.506006 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:26.508853 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:26.509606 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:26.509613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:26.509615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:26.509750 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:26.510151 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:26.510154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:26.510156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:26.510587 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:26.510814 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:26.511185 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 07:12:26.511358 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260620 07:12:26.512566 19707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:26.548848 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.548306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42301 (local address 127.0.0.1:47394)
0620 07:12:26.548536 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:26.548643 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 07:12:26.549999 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.549529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42301 (local address 127.0.0.1:47410)
0620 07:12:26.549728 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:26.549839 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 07:12:26.600693 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 19575
I20260620 07:12:26.617079 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 19195
I20260620 07:12:26.618456 19258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:26.739655 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.739024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42301 (local address 127.0.0.1:47412)
0620 07:12:26.739324 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:26.739455 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260620 07:12:26.739691 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 19195
I20260620 07:12:26.763702 6357 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 07:12:27.098630 6357 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 07:12:27.098894 6357 mini_ranger.cc:66] Stopping Ranger...
I20260620 07:12:27.444276 6357 mini_ranger.cc:68] Stopped Ranger
2026-06-20 07:12:27.444 UTC [18247] LOG: received smart shutdown request
2026-06-20 07:12:27.447 UTC [18247] LOG: background worker "logical replication launcher" (PID 18256) exited with exit code 1
2026-06-20 07:12:27.447 UTC [18251] LOG: shutting down
2026-06-20 07:12:27.603 UTC [18247] LOG: database system is shut down
2026-06-20T07:12:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51891 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:12:27 dist-test-slave-4n6h krb5kdc[19773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:12:27 dist-test-slave-4n6h krb5kdc[19773](info): set up 2 sockets
Jun 20 07:12:27 dist-test-slave-4n6h krb5kdc[19773](info): commencing operation
krb5kdc: starting...
W20260620 07:12:28.826610 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:12:28 dist-test-slave-4n6h krb5kdc[19773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:12:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:12:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:12:29.045218 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:35609
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:37499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:35609,127.6.53.125:34185,127.6.53.124:36547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:29.358843 19786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:29.359109 19786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:29.359277 19786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:29.366240 19786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:29.366348 19786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:29.366465 19786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:29.366561 19786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:29.366643 19786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:29.376129 19786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:37499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:35609,127.6.53.125:34185,127.6.53.124:36547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:35609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:29.377414 19786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:29.378677 19786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:29.390034 19791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:29.390003 19792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:29.390812 19794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:29.391742 19786 server_base.cc:1034] running on GCE node
I20260620 07:12:29.392434 19786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:29.393777 19786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:29.394969 19786 hybrid_clock.cc:648] HybridClock initialized: now 1781939549394917 us; error 48 us; skew 500 ppm
I20260620 07:12:29.395313 19786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:29 dist-test-slave-4n6h krb5kdc[19773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939549, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:29.398233 19786 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:12:29.399677 19786 webserver.cc:460] Webserver started at http://127.6.53.126:33963/ using document root <none> and password file <none>
I20260620 07:12:29.400352 19786 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:29.400496 19786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:29.400812 19786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:29.403744 19786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "4717c732dc664e559fae290f58b57124"
format_stamp: "Formatted at 2026-06-20 07:12:29 on dist-test-slave-4n6h"
server_key: "8efe6f65eb241258795520f8c7ada38e"
server_key_iv: "3299d55e78239028070d5f14be196c47"
server_key_version: "encryptionkey@0"
I20260620 07:12:29.404560 19786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "4717c732dc664e559fae290f58b57124"
format_stamp: "Formatted at 2026-06-20 07:12:29 on dist-test-slave-4n6h"
server_key: "8efe6f65eb241258795520f8c7ada38e"
server_key_iv: "3299d55e78239028070d5f14be196c47"
server_key_version: "encryptionkey@0"
I20260620 07:12:29.409508 19786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260620 07:12:29.412736 19801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:29.413983 19786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:29.414168 19786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "4717c732dc664e559fae290f58b57124"
format_stamp: "Formatted at 2026-06-20 07:12:29 on dist-test-slave-4n6h"
server_key: "8efe6f65eb241258795520f8c7ada38e"
server_key_iv: "3299d55e78239028070d5f14be196c47"
server_key_version: "encryptionkey@0"
I20260620 07:12:29.414408 19786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:29.429170 19786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:29.433029 19786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:29.433305 19786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:29.451295 19786 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:35609
I20260620 07:12:29.451324 19852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:35609 every 8 connection(s)
I20260620 07:12:29.452615 19786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:12:29.455355 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 19786
I20260620 07:12:29.455580 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:12:29.456374 6357 external_mini_cluster.cc:1428] Setting key a4d4454fc10e3872537f0ad2ed8789a4
I20260620 07:12:29.456681 19853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:29.465878 19853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.53.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.125@KRBTEST.COM" created.
Jun 20 07:12:29 dist-test-slave-4n6h krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939549, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:12:29.494529 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.468928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:35609 (local address 127.0.0.1:41504)
0620 07:12:29.470042 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.470073 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:29.471785 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:29.472556 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:29.472562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:29.473247 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:29.473556 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:29.473575 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.476087 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.476091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:29.477130 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:29.477141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.478496 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.478501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:29.478614 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:29.483843 (+ 5229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:29.483890 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:29.486583 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:29.490748 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:29.490758 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:29.490767 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:29.491145 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:29.491625 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:29.491628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:29.491630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:29.491767 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:29.492714 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:29.492719 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:29.492892 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 07:12:29.493095 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 07:12:29.493469 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":675,"threads_started":1}
I20260620 07:12:29.494617 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.468927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.6.53.126:35609)
0620 07:12:29.469732 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:29.469753 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:29.470195 (+ 442us) server_negotiation.cc:407] Connection header received
0620 07:12:29.472130 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:29.472135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:29.472338 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:29.472541 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:29.473808 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.475929 (+ 2121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.477278 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.478200 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.478426 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:29.486824 (+ 8398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:29.486833 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:29.486847 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:29.486877 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:29.490613 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:29.491280 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:29.491292 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:29.491298 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:29.491509 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:29.491867 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:29.491874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:29.491881 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:29.492572 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:29.492764 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:29.493279 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 07:12:29.493683 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":565,"thread_start_us":358,"threads_started":1}
I20260620 07:12:29.497473 19853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:12:29.498585 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.498195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:34185 (local address 127.0.0.1:55692)
0620 07:12:29.498376 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.498490 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 07:12:29.498832 19803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.53.125:34185: connect: Connection refused (error 111)
I20260620 07:12:29.500425 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.500042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:34185 (local address 127.0.0.1:55708)
0620 07:12:29.500224 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.500288 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 07:12:29.500624 19853 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.125:34185: Network error: Client connection negotiation failed: client connection to 127.6.53.125:34185: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.125.keytab.
Entry for principal kudu/127.6.53.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.125.keytab.
I20260620 07:12:29.509923 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.125:34185
--webserver_interface=127.6.53.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.125.keytab
--principal=kudu/127.6.53.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:37499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:35609,127.6.53.125:34185,127.6.53.124:36547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:12:29.549960 19853 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } attempt: 1
I20260620 07:12:29.551205 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.550738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:34185 (local address 127.0.0.1:55720)
0620 07:12:29.550949 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.551050 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 07:12:29.552306 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.551937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:34185 (local address 127.0.0.1:55726)
0620 07:12:29.552095 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.552160 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 07:12:29.552507 19853 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.125:34185: Network error: Client connection negotiation failed: client connection to 127.6.53.125:34185: connect: Connection refused (error 111)
I20260620 07:12:29.620769 19853 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } attempt: 2
I20260620 07:12:29.622006 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.621462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:34185 (local address 127.0.0.1:55734)
0620 07:12:29.621746 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.621844 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 07:12:29.623227 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.622831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:34185 (local address 127.0.0.1:55736)
0620 07:12:29.622985 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.623082 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 07:12:29.623438 19853 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.125:34185: Network error: Client connection negotiation failed: client connection to 127.6.53.125:34185: connect: Connection refused (error 111)
I20260620 07:12:29.714679 19853 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } attempt: 3
I20260620 07:12:29.715780 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.715347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:34185 (local address 127.0.0.1:55744)
0620 07:12:29.715547 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.715644 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 07:12:29.716766 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.716423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:34185 (local address 127.0.0.1:55752)
0620 07:12:29.716565 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.716648 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 07:12:29.716964 19853 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.125:34185: Network error: Client connection negotiation failed: client connection to 127.6.53.125:34185: connect: Connection refused (error 111)
W20260620 07:12:29.826726 19860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:29.827045 19860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:29.827203 19860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:29.834422 19860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:29.834558 19860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:29.834666 19860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:29.834770 19860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:29.834856 19860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:29.848352 19860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:37499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:35609,127.6.53.125:34185,127.6.53.124:36547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.125.keytab
--principal=kudu/127.6.53.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.125:34185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.53.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:29.849648 19860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:29.850926 19860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:12:29.860180 19853 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } attempt: 4
I20260620 07:12:29.861481 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.860878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:34185 (local address 127.0.0.1:55764)
0620 07:12:29.861195 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.861309 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
W20260620 07:12:29.862617 19865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:29.862991 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.862532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:34185 (local address 127.0.0.1:55780)
0620 07:12:29.862751 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.862869 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:34185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 07:12:29.863329 19853 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.125:34185: Network error: Client connection negotiation failed: client connection to 127.6.53.125:34185: connect: Connection refused (error 111)
W20260620 07:12:29.863435 19866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:29.863759 19868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:29.863914 19860 server_base.cc:1034] running on GCE node
I20260620 07:12:29.864651 19860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:29.865942 19860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:29.867161 19860 hybrid_clock.cc:648] HybridClock initialized: now 1781939549867123 us; error 49 us; skew 500 ppm
I20260620 07:12:29.867517 19860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:29 dist-test-slave-4n6h krb5kdc[19773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939549, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:29.870659 19860 init.cc:377] Logged in from keytab as kudu/127.6.53.125@KRBTEST.COM (short username kudu)
I20260620 07:12:29.872129 19860 webserver.cc:460] Webserver started at http://127.6.53.125:34769/ using document root <none> and password file <none>
I20260620 07:12:29.872833 19860 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:29.872992 19860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:29.873308 19860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:29.876392 19860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/data/instance:
uuid: "02c99bc5d2eb43a49b145488b300ea2e"
format_stamp: "Formatted at 2026-06-20 07:12:29 on dist-test-slave-4n6h"
server_key: "73df5d257d1618a28d855112d79fee91"
server_key_iv: "2fd9d336434e73bfcceab3c0ec54447c"
server_key_version: "encryptionkey@0"
I20260620 07:12:29.877244 19860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/wal/instance:
uuid: "02c99bc5d2eb43a49b145488b300ea2e"
format_stamp: "Formatted at 2026-06-20 07:12:29 on dist-test-slave-4n6h"
server_key: "73df5d257d1618a28d855112d79fee91"
server_key_iv: "2fd9d336434e73bfcceab3c0ec54447c"
server_key_version: "encryptionkey@0"
I20260620 07:12:29.882444 19860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 07:12:29.885795 19875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:29.887122 19860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:12:29.887318 19860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/wal
uuid: "02c99bc5d2eb43a49b145488b300ea2e"
format_stamp: "Formatted at 2026-06-20 07:12:29 on dist-test-slave-4n6h"
server_key: "73df5d257d1618a28d855112d79fee91"
server_key_iv: "2fd9d336434e73bfcceab3c0ec54447c"
server_key_version: "encryptionkey@0"
I20260620 07:12:29.887526 19860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:29.899680 19860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:29.904177 19860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:29.904482 19860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:29.933385 19860 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.125:34185
I20260620 07:12:29.933415 19926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.125:34185 every 8 connection(s)
I20260620 07:12:29.934903 19860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/data/info.pb
I20260620 07:12:29.938565 19927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:29.943140 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 19860
I20260620 07:12:29.943347 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-1/wal/instance
I20260620 07:12:29.943899 6357 external_mini_cluster.cc:1428] Setting key 59f5770f573c3288a7af7b38fdb5c4bb
I20260620 07:12:29.945649 19927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:12:29 dist-test-slave-4n6h krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939549, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.125@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.53.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.124@KRBTEST.COM" created.
I20260620 07:12:29.968142 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.949005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.6.53.126:35609)
0620 07:12:29.949231 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:29.949237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:29.950478 (+ 1241us) server_negotiation.cc:407] Connection header received
0620 07:12:29.952308 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:29.952313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:29.952382 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:29.952476 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:29.954443 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.955095 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.957514 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.957936 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.958034 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:29.962194 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:29.962200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:29.962203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:29.962235 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:29.964974 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:29.965776 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:29.965781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:29.965783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:29.965903 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:29.966330 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:29.966335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:29.966338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:29.966646 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:29.966802 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:29.967584 (+ 782us) server_negotiation.cc:299] Negotiation successful
0620 07:12:29.967808 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:12:29.969132 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.949446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:35609 (local address 127.0.0.1:41506)
0620 07:12:29.950300 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.950352 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:29.952001 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:29.952711 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:29.952724 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:29.953473 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:29.954271 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:29.954300 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.955272 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.955280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:29.957375 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:29.957393 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.958171 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.958181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:29.958519 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:29.959293 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:29.959327 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:29.961960 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:29.965123 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:29.965132 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:29.965148 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:29.965646 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:29.966013 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:29.966019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:29.966023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:29.966195 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:29.966773 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:29.966780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:29.967129 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 07:12:29.967433 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 07:12:29.967899 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":390,"threads_started":1}
I20260620 07:12:29.971087 19927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:12:29 dist-test-slave-4n6h krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939549, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.125@KRBTEST.COM for kudu/127.6.53.125@KRBTEST.COM
I20260620 07:12:29.986721 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.971901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:34185 (local address 127.0.0.1:55796)
0620 07:12:29.972129 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.972149 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:29.972255 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:29.973343 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:29.973350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:29.973781 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:29.973977 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:29.973990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.975621 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.975631 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:29.977173 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:29.977191 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.977981 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.977993 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:29.978272 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:29.978843 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:29.978869 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:29.981277 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:29.984353 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:29.984359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:29.984363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:29.984684 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:29.985033 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:29.985039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:29.985045 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:29.985222 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:29.986016 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:29.986019 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:29.986135 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 07:12:29.986238 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 07:12:29.986484 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 07:12:29.987211 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.972164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.6.53.125:34185)
0620 07:12:29.972818 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:29.972827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:29.972856 (+ 29us) server_negotiation.cc:407] Connection header received
0620 07:12:29.972910 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:29.972919 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:29.973152 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:29.973329 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:29.974157 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.975491 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.977316 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.977821 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.977922 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:29.981495 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:29.981506 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:29.981518 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:29.981544 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:29.984202 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:29.984806 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:29.984815 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:29.984818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:29.984930 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:29.985352 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:29.985361 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:29.985366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:29.985888 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:29.986100 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:29.986459 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 07:12:29.986745 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":350,"threads_started":1}
I20260620 07:12:29.989735 19927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:12:29.991092 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.990504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55018)
0620 07:12:29.990731 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.990869 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 07:12:29.991355 19877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.124.keytab.
Entry for principal kudu/127.6.53.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.124.keytab.
I20260620 07:12:29.995667 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.995213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55028)
0620 07:12:29.995420 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.995524 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 07:12:29.995908 19927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:36547: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
I20260620 07:12:29.999087 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.124:36547
--webserver_interface=127.6.53.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.124.keytab
--principal=kudu/127.6.53.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:37499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:35609,127.6.53.125:34185,127.6.53.124:36547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:12:30.045284 19927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } attempt: 1
I20260620 07:12:30.046778 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.046124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55034)
0620 07:12:30.046413 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.046560 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 07:12:30.047807 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.047420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55040)
0620 07:12:30.047579 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.047673 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 07:12:30.048081 19927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:36547: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
I20260620 07:12:30.116393 19927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } attempt: 2
I20260620 07:12:30.117589 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.117071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55046)
0620 07:12:30.117280 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.117400 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 07:12:30.118629 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.118242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55050)
0620 07:12:30.118417 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.118496 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 07:12:30.118846 19927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:36547: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
I20260620 07:12:30.162608 19853 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } attempt: 5
Jun 20 07:12:30 dist-test-slave-4n6h krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939549, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for kudu/127.6.53.125@KRBTEST.COM
I20260620 07:12:30.177664 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.163314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:34185 (local address 127.0.0.1:55798)
0620 07:12:30.163566 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.163590 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:30.163725 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:30.164087 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:30.164092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:30.164657 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:30.164924 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.164941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.165972 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.165980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.167662 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.167683 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.168445 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.168450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.168631 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.169293 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:30.169316 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:30.171604 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:30.174969 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:30.174977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:30.174981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:30.175488 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:30.175890 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:30.175896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:30.175900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:30.176050 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:30.176626 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:30.176631 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:30.176816 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 07:12:30.176969 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 07:12:30.177248 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 07:12:30.177664 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.163542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55798 (local address 127.6.53.125:34185)
0620 07:12:30.163706 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:30.163710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:30.163729 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:12:30.163865 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:30.163869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:30.163944 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:30.164025 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:30.165116 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.165810 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.167816 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.168267 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.168419 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.171790 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:30.171794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:30.171798 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:30.171835 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:30.174812 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.175620 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.175628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.175632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.175770 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.176165 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.176173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.176176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.176497 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:30.176665 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:30.177072 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 07:12:30.177306 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:12:30.178838 19853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:12:30.179864 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.179445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55056)
0620 07:12:30.179618 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.179721 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 07:12:30.180888 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.180572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55066)
0620 07:12:30.180715 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.180781 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 07:12:30.181176 19853 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:36547: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
I20260620 07:12:30.210100 19927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } attempt: 3
I20260620 07:12:30.211323 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.210793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55080)
0620 07:12:30.211022 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.211126 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 07:12:30.212519 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.211983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55096)
0620 07:12:30.212233 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.212348 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260620 07:12:30.212756 19927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:36547: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
I20260620 07:12:30.232446 19853 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } attempt: 1
I20260620 07:12:30.233515 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.233098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55108)
0620 07:12:30.233274 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.233352 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 07:12:30.234539 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.234193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55114)
0620 07:12:30.234372 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.234444 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260620 07:12:30.234798 19853 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:36547: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
I20260620 07:12:30.303025 19853 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } attempt: 2
I20260620 07:12:30.304206 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.303703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55122)
0620 07:12:30.303916 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.304037 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 07:12:30.305286 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.304946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55128)
0620 07:12:30.305087 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.305156 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 07:12:30.305576 19853 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:36547: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
W20260620 07:12:30.321424 19934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:30.321667 19934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:30.321851 19934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:30.328896 19934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:30.328985 19934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:30.329062 19934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:30.329152 19934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:30.329217 19934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:30.338632 19934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:37499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:35609,127.6.53.125:34185,127.6.53.124:36547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.124.keytab
--principal=kudu/127.6.53.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.124:36547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.53.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:30.339912 19934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:30.341145 19934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:30.353551 19939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:30.353492 19942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:30.353524 19940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:30.354533 19934 server_base.cc:1034] running on GCE node
I20260620 07:12:30.355295 19934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:30.355983 19927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } attempt: 4
I20260620 07:12:30.356868 19934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:30.357321 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.356596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55130)
0620 07:12:30.356836 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.356978 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 07:12:30.358085 19934 hybrid_clock.cc:648] HybridClock initialized: now 1781939550358033 us; error 45 us; skew 500 ppm
I20260620 07:12:30.358536 19934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:12:30.358594 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.358064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55146)
0620 07:12:30.358357 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.358449 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260620 07:12:30.358875 19927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:36547: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Jun 20 07:12:30 dist-test-slave-4n6h krb5kdc[19773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939550, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:30.361882 19934 init.cc:377] Logged in from keytab as kudu/127.6.53.124@KRBTEST.COM (short username kudu)
I20260620 07:12:30.363502 19934 webserver.cc:460] Webserver started at http://127.6.53.124:35683/ using document root <none> and password file <none>
I20260620 07:12:30.364177 19934 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:30.364305 19934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:30.364620 19934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:30.367537 19934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/data/instance:
uuid: "c6d4c84824654cad995663d1f245967d"
format_stamp: "Formatted at 2026-06-20 07:12:30 on dist-test-slave-4n6h"
server_key: "39f8166af3576f0f3a0c887b5945b048"
server_key_iv: "deea26b7e6bd9cd338f7c14e000959b3"
server_key_version: "encryptionkey@0"
I20260620 07:12:30.368357 19934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/wal/instance:
uuid: "c6d4c84824654cad995663d1f245967d"
format_stamp: "Formatted at 2026-06-20 07:12:30 on dist-test-slave-4n6h"
server_key: "39f8166af3576f0f3a0c887b5945b048"
server_key_iv: "deea26b7e6bd9cd338f7c14e000959b3"
server_key_version: "encryptionkey@0"
I20260620 07:12:30.373216 19934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:12:30.376737 19949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:30.378060 19934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:30.378276 19934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/wal
uuid: "c6d4c84824654cad995663d1f245967d"
format_stamp: "Formatted at 2026-06-20 07:12:30 on dist-test-slave-4n6h"
server_key: "39f8166af3576f0f3a0c887b5945b048"
server_key_iv: "deea26b7e6bd9cd338f7c14e000959b3"
server_key_version: "encryptionkey@0"
I20260620 07:12:30.378505 19934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:30.402135 19934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:30.406201 19934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:30.406576 19934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:30.411815 19853 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } attempt: 3
I20260620 07:12:30.413113 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.412445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55156)
0620 07:12:30.412749 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.412870 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 07:12:30.414536 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.413981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55164)
0620 07:12:30.414204 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.414358 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 07:12:30.414793 19853 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:36547: Network error: Client connection negotiation failed: client connection to 127.6.53.124:36547: connect: Connection refused (error 111)
I20260620 07:12:30.426093 19934 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.124:36547
I20260620 07:12:30.426115 20000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.124:36547 every 8 connection(s)
I20260620 07:12:30.427501 19934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/data/info.pb
I20260620 07:12:30.429287 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 19934
I20260620 07:12:30.429488 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939352012798-6357-0/minicluster-data/master-2/wal/instance
I20260620 07:12:30.429977 6357 external_mini_cluster.cc:1428] Setting key 13d23c40d97d45251026a251736f9a62
I20260620 07:12:30.431473 20001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:30.440495 20001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:12:30 dist-test-slave-4n6h krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.124@KRBTEST.COM
Jun 20 07:12:30 dist-test-slave-4n6h krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939550, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.124@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:12:30.455112 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.431909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55178 (local address 127.6.53.124:36547)
0620 07:12:30.432759 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:30.432771 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:30.432812 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:12:30.434487 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:30.434508 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:30.434881 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:30.435229 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:30.437010 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.438542 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.440494 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.441287 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.441516 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.446463 (+ 4947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:30.446471 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:30.446480 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:30.446516 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:30.450443 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.451184 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.451192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.451207 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.451379 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.451843 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.451849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.451852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.452544 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:30.452829 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:30.453292 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 07:12:30.453706 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":403,"threads_started":1}
I20260620 07:12:30.460325 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.442975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:35609 (local address 127.0.0.1:41512)
0620 07:12:30.443789 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.443836 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:30.443979 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:30.444397 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:30.444402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:30.444929 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:30.445181 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.445190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.446693 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.446699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.448405 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.448416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.449229 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.449235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.449413 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.450025 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:30.450051 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:30.452829 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:30.456911 (+ 4082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:30.456921 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:30.456928 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:30.457873 (+ 945us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:30.458387 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:30.458392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:30.458395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:30.458566 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:30.459231 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:30.459240 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:30.459425 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 07:12:30.459614 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 07:12:30.459918 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":447,"threads_started":1}
I20260620 07:12:30.460417 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.442995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.6.53.126:35609)
0620 07:12:30.443247 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:30.443255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:30.443939 (+ 684us) server_negotiation.cc:407] Connection header received
0620 07:12:30.444108 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:30.444114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:30.444214 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:30.444340 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:30.445431 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.446539 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.448555 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.449035 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.449138 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.453034 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:30.453042 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:30.453051 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:30.453097 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:30.456768 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.458000 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.458007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.458016 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.458223 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.458720 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.458736 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.458741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.459078 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:30.459247 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:30.459761 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 07:12:30.460002 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:12:30.462198 20001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:12:30 dist-test-slave-4n6h krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
Jun 20 07:12:30 dist-test-slave-4n6h krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939550, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.124@KRBTEST.COM for kudu/127.6.53.125@KRBTEST.COM
I20260620 07:12:30.474747 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.458427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.6.53.126:35609)
0620 07:12:30.458976 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:30.458982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:30.459004 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:12:30.459078 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:30.459084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:30.459156 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:30.459267 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:30.461072 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.462230 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.463986 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.464380 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.464506 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.469220 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:30.469232 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:30.469235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:30.469264 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:30.472017 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.472715 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.472721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.472728 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.472869 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.473241 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.473247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.473250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.473519 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:30.473792 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:30.474066 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 07:12:30.474348 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":221,"threads_started":1}
I20260620 07:12:30.475633 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.463013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:34185 (local address 127.0.0.1:55808)
0620 07:12:30.463273 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.463302 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:30.463383 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:30.463662 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:30.463665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:30.463959 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:30.464081 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.464087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.465074 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.465078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.466610 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.466627 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.467320 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.467328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.467576 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.468141 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:30.468156 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:30.470190 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:30.473168 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:30.473173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:30.473177 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:30.473534 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:30.473912 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:30.473918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:30.473921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:30.474063 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:30.474681 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:30.474697 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:30.474867 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 07:12:30.475021 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 07:12:30.475288 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260620 07:12:30.475678 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.463001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55808 (local address 127.6.53.125:34185)
0620 07:12:30.463169 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:30.463173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:30.463382 (+ 209us) server_negotiation.cc:407] Connection header received
0620 07:12:30.463459 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:30.463472 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:30.463545 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:30.463660 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:30.464230 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.464911 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.466760 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.467167 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.467261 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.470406 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:30.470411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:30.470414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:30.470445 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:30.473025 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.473652 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.473659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.473662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.473803 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.474171 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.474178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.474181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.474554 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:30.474703 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:30.475111 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 07:12:30.475324 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:12:30.477036 20001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:12:30 dist-test-slave-4n6h krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939550, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.124@KRBTEST.COM for kudu/127.6.53.124@KRBTEST.COM
Jun 20 07:12:30 dist-test-slave-4n6h krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.125@KRBTEST.COM
I20260620 07:12:30.489885 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.477779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55190 (local address 127.6.53.124:36547)
0620 07:12:30.477929 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:30.477936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:30.477953 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:12:30.478118 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:30.478123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:30.478191 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:30.478330 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:30.479007 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.479691 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.480840 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.481288 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.481434 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.484885 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:30.484892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:30.484897 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:30.484938 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:30.487397 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.488003 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.488011 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.488015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.488133 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.488491 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.488496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.488499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.488749 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:30.488894 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:30.489228 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 07:12:30.489456 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":59}
I20260620 07:12:30.489962 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.477702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55190)
0620 07:12:30.477875 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.477893 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:30.477994 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:30.478344 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:30.478348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:30.478739 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:30.478865 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.478873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.479821 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.479828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.480696 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.480707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.481447 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.481453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.481705 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.482344 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:30.482363 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:30.484701 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:30.487525 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:30.487533 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:30.487537 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:30.487875 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:30.488252 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:30.488256 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:30.488260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:30.488382 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:30.488866 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:30.488871 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:30.489011 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 07:12:30.489157 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 07:12:30.489597 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":174}
I20260620 07:12:30.491034 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.476440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.6.53.125:34185)
0620 07:12:30.476596 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:30.476604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:30.476621 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:12:30.476807 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:30.476812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:30.476887 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:30.477005 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:30.478361 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.479055 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.480266 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.480727 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.480868 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.485409 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:30.485416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:30.485422 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:30.485461 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:30.488166 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.488848 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.488854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.488857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.488990 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.489338 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.489344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.489347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.489588 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:30.489864 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:30.490440 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 07:12:30.490688 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:12:30.495466 20001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d: Bootstrap starting.
I20260620 07:12:30.496166 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:12:30.498593 20001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:30.499604 20001 log.cc:824] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:30.503821 20001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d: No bootstrap required, opened a new log
I20260620 07:12:30.517464 20001 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } }
I20260620 07:12:30.518083 20001 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:30.518258 20001 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d4c84824654cad995663d1f245967d, State: Initialized, Role: FOLLOWER
I20260620 07:12:30.519196 20001 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } }
I20260620 07:12:30.525651 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.501093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55834 (local address 127.6.53.125:34185)
0620 07:12:30.501286 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:30.501294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:30.501956 (+ 662us) server_negotiation.cc:407] Connection header received
0620 07:12:30.502216 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:30.502221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:30.502342 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:30.502420 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:30.505452 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.506958 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.509758 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.510670 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.510809 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.517126 (+ 6317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:30.517133 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:30.517138 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:30.517181 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:30.521040 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.522301 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.522310 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.522317 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.522503 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.523100 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.523108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.523114 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.523589 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:30.523904 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:30.524638 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 07:12:30.525134 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:12:30.526582 20015 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } } }
I20260620 07:12:30.526950 20015 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:12:30.528949 20001 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:30.530892 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.501136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.6.53.126:35609)
0620 07:12:30.501345 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:30.501368 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:30.512515 (+ 11147us) server_negotiation.cc:407] Connection header received
0620 07:12:30.513006 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:30.513012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:30.513116 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:30.513203 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:30.515463 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.516502 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.518858 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.519512 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.519608 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.523221 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:30.523229 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:30.523337 (+ 108us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:30.523382 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:30.527003 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.527728 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.527745 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.527748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.527903 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.528377 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.528382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.528385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.528901 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:30.529454 (+ 553us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:30.530058 (+ 604us) server_negotiation.cc:299] Negotiation successful
0620 07:12:30.530389 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:12:30.534192 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.502425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.6.53.124:36547)
0620 07:12:30.504133 (+ 1708us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:30.504142 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:30.515707 (+ 11565us) server_negotiation.cc:407] Connection header received
0620 07:12:30.515964 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:30.515969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:30.516051 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:30.516157 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:30.518511 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.519846 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.522316 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.523040 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.523144 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.526803 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:30.526810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:30.526815 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:30.526860 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:30.530680 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.531447 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.531452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.531456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.531773 (+ 317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.532278 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.532284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.532287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.532725 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:30.533069 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:30.533555 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 07:12:30.533777 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1532}
W20260620 07:12:30.544898 20026 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:12:30.545147 20026 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:12:30.538211 19934 master_runner.cc:186] Error getting master registration for 127.6.53.126:35609: OK, instance_id { permanent_uuid: "4717c732dc664e559fae290f58b57124" instance_seqno: 1781939549446114 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:12:30.548772 19934 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:12:30.592039 19853 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } attempt: 4
Jun 20 07:12:30 dist-test-slave-4n6h krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939549, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for kudu/127.6.53.124@KRBTEST.COM
I20260620 07:12:30.606905 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.592915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.6.53.124:36547)
0620 07:12:30.593127 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:30.593133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:30.593174 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:12:30.593254 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:30.593260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:30.593356 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:30.593465 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:30.594231 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.595475 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.596959 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.597692 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.597826 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.601441 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:30.601446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:30.601449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:30.601478 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:30.604275 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.605049 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.605052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.605055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.605189 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.605563 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.605569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.605571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.605835 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:30.605940 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:30.606391 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 07:12:30.606553 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:12:30.607062 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.592683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55208)
0620 07:12:30.592914 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.592936 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:30.593074 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:30.593505 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:30.593509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:30.593904 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:30.594076 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.594085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.595616 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.595623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.596803 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.596816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.597871 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.597879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.598028 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.598797 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:30.598817 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:30.601274 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:30.604404 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:30.604410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:30.604414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:30.604934 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:30.605296 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:30.605301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:30.605305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:30.605463 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:30.605942 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:30.605947 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:30.606134 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 07:12:30.606317 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 07:12:30.606648 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 07:12:30.611910 19853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124: Bootstrap starting.
I20260620 07:12:30.614960 19853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:30.615962 19853 log.cc:824] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:30.618636 19853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124: No bootstrap required, opened a new log
I20260620 07:12:30.626605 19853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } }
I20260620 07:12:30.626925 19853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:30.627050 19853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4717c732dc664e559fae290f58b57124, State: Initialized, Role: FOLLOWER
I20260620 07:12:30.627602 19853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } }
I20260620 07:12:30.628500 20029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } } }
I20260620 07:12:30.628875 20029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:12:30.629746 19853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 07:12:30.639336 20040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:12:30.639463 20040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:12:30.638990 19786 master_runner.cc:186] Error getting master registration for 127.6.53.125:34185: OK, instance_id { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" instance_seqno: 1781939549927794 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:12:30.649896 19786 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:12:30.658119 19927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } attempt: 5
W20260620 07:12:30.658598 19955 tablet.cc:2369] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 07:12:30 dist-test-slave-4n6h krb5kdc[19773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939549, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.125@KRBTEST.COM for kudu/127.6.53.124@KRBTEST.COM
I20260620 07:12:30.676981 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.658737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:36547 (local address 127.0.0.1:55210)
0620 07:12:30.659106 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.659128 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:30.659245 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:30.659996 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:30.660002 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:30.660542 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:30.660785 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.660798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.662233 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.662239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.663727 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.663744 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.665001 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.665008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.665284 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.666049 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:30.666076 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:30.669104 (+ 3028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:30.673711 (+ 4607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:30.673717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:30.673721 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:30.674281 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:30.674784 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:30.674789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:30.674793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:30.674986 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:30.675782 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:30.675788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:30.675964 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 07:12:30.676143 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 07:12:30.676448 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260620 07:12:30.678422 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.659460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.6.53.124:36547)
0620 07:12:30.659653 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:30.659659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:30.659679 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:12:30.659745 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:30.659751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:30.659831 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:30.659934 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:30.660957 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.662078 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.663929 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.664808 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.664960 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.669308 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:30.669316 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:30.669321 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:30.669363 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:30.673519 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.674451 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.674457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.674460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.674608 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:30.675141 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:30.675147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:30.675150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:30.675533 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:30.675710 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:30.677452 (+ 1742us) server_negotiation.cc:299] Negotiation successful
0620 07:12:30.677944 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:12:30.684393 19927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e: Bootstrap starting.
W20260620 07:12:30.686087 19807 tablet.cc:2369] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:12:30.688539 19927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:30.689816 19927 log.cc:824] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:30.693146 19927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e: No bootstrap required, opened a new log
I20260620 07:12:30.703938 19927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } }
I20260620 07:12:30.704341 19927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:30.704459 19927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c99bc5d2eb43a49b145488b300ea2e, State: Initialized, Role: FOLLOWER
I20260620 07:12:30.705080 19927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } }
I20260620 07:12:30.706067 20043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } } }
I20260620 07:12:30.706622 20043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:12:30.707463 19927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:30.724414 19860 master_runner.cc:263] No leader master found from master 02c99bc5d2eb43a49b145488b300ea2e
I20260620 07:12:30.724560 19860 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 07:12:30.727324 20054 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:12:30.727550 20054 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:12:30.785902 20029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:30.786404 20029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } }
I20260620 07:12:30.788321 20029 leader_election.cc:288] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02c99bc5d2eb43a49b145488b300ea2e (127.6.53.125:34185), c6d4c84824654cad995663d1f245967d (127.6.53.124:36547)
I20260620 07:12:30.789669 19902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4717c732dc664e559fae290f58b57124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c99bc5d2eb43a49b145488b300ea2e" is_pre_election: true
I20260620 07:12:30.790392 19902 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4717c732dc664e559fae290f58b57124 in term 0.
I20260620 07:12:30.791625 19976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4717c732dc664e559fae290f58b57124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d4c84824654cad995663d1f245967d" is_pre_election: true
I20260620 07:12:30.792349 19976 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4717c732dc664e559fae290f58b57124 in term 0.
I20260620 07:12:30.793169 19803 leader_election.cc:302] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c99bc5d2eb43a49b145488b300ea2e, 4717c732dc664e559fae290f58b57124; no voters:
I20260620 07:12:30.798301 20029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:12:30.798475 20029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:12:30.798593 20029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:30.802495 20029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } }
I20260620 07:12:30.803812 19902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4717c732dc664e559fae290f58b57124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c99bc5d2eb43a49b145488b300ea2e"
I20260620 07:12:30.804072 19902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:30.808235 19902 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4717c732dc664e559fae290f58b57124 in term 1.
I20260620 07:12:30.808931 19803 leader_election.cc:302] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c99bc5d2eb43a49b145488b300ea2e, 4717c732dc664e559fae290f58b57124; no voters:
I20260620 07:12:30.812557 20029 leader_election.cc:288] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [CANDIDATE]: Term 1 election: Requested vote from peers 02c99bc5d2eb43a49b145488b300ea2e (127.6.53.125:34185), c6d4c84824654cad995663d1f245967d (127.6.53.124:36547)
I20260620 07:12:30.813257 20029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:30.813720 20029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [term 1 LEADER]: Becoming Leader. State: Replica: 4717c732dc664e559fae290f58b57124, State: Running, Role: LEADER
I20260620 07:12:30.814685 20029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } }
I20260620 07:12:30.826956 19976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4717c732dc664e559fae290f58b57124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d4c84824654cad995663d1f245967d"
I20260620 07:12:30.827241 19976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:30.831367 19976 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4717c732dc664e559fae290f58b57124 in term 1.
I20260620 07:12:30.834105 20056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4717c732dc664e559fae290f58b57124. Latest consensus state: current_term: 1 leader_uuid: "4717c732dc664e559fae290f58b57124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } } }
I20260620 07:12:30.834605 20056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:30.850824 20058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:12:30.853173 20058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:12:30.869225 19976 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d [term 1 FOLLOWER]: Refusing update from remote peer 4717c732dc664e559fae290f58b57124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:30.870584 19902 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e [term 1 FOLLOWER]: Refusing update from remote peer 4717c732dc664e559fae290f58b57124: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:30.871903 20056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:30.873442 20056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:12:30.907094 19881 tablet.cc:2369] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:12:30.923833 20043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4717c732dc664e559fae290f58b57124. Latest consensus state: current_term: 1 leader_uuid: "4717c732dc664e559fae290f58b57124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } } }
I20260620 07:12:30.924242 20043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:12:30.928301 20059 mvcc.cc:205] Tried to move back new op lower bound from 7298824400347398144 to 7298824400169275392. Current Snapshot: MvccSnapshot[applied={T|T < 7298824400347398144}]
I20260620 07:12:30.955380 20061 mvcc.cc:205] Tried to move back new op lower bound from 7298824400347398144 to 7298824400169275392. Current Snapshot: MvccSnapshot[applied={T|T < 7298824400347398144}]
I20260620 07:12:30.974740 20015 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4717c732dc664e559fae290f58b57124. Latest consensus state: current_term: 1 leader_uuid: "4717c732dc664e559fae290f58b57124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } } }
I20260620 07:12:30.975122 20015 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:12:30.985772 20029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4717c732dc664e559fae290f58b57124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } } }
I20260620 07:12:30.986123 20029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:30.986495 20029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4717c732dc664e559fae290f58b57124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } } }
I20260620 07:12:30.986756 20029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:30.988018 20058 catalog_manager.cc:1347] Generated new cluster ID: 0e703deed75446409f378ca8c9484d7a
I20260620 07:12:30.988119 20058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:12:30.988533 20043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4717c732dc664e559fae290f58b57124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } } }
I20260620 07:12:30.988780 20043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:12:30.998023 20015 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4717c732dc664e559fae290f58b57124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4717c732dc664e559fae290f58b57124" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35609 } } peers { permanent_uuid: "02c99bc5d2eb43a49b145488b300ea2e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 34185 } } peers { permanent_uuid: "c6d4c84824654cad995663d1f245967d" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 36547 } } }
I20260620 07:12:30.998299 20015 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:12:31.049449 20058 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:12:31.051335 20058 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:12:31.066903 20058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4717c732dc664e559fae290f58b57124: Generated new TSK 0
I20260620 07:12:31.068187 20058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:12:31.547453 20026 catalog_manager.cc:1259] Loaded cluster ID: 0e703deed75446409f378ca8c9484d7a
I20260620 07:12:31.547592 20026 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d: loading cluster ID for follower catalog manager: success
I20260620 07:12:31.550377 20026 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d: acquiring CA information for follower catalog manager: success
I20260620 07:12:31.552045 20026 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c6d4c84824654cad995663d1f245967d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:12:31.729736 20054 catalog_manager.cc:1259] Loaded cluster ID: 0e703deed75446409f378ca8c9484d7a
I20260620 07:12:31.729882 20054 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e: loading cluster ID for follower catalog manager: success
I20260620 07:12:31.732493 20054 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e: acquiring CA information for follower catalog manager: success
I20260620 07:12:31.734421 20054 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 02c99bc5d2eb43a49b145488b300ea2e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:12:32.163786 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 19786
I20260620 07:12:32.164991 19848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:32.272967 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 19786
I20260620 07:12:32.293751 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 19860
I20260620 07:12:32.294826 19922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:32.399605 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 19860
I20260620 07:12:32.419875 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 19934
I20260620 07:12:32.420924 19996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:32.526381 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 19934
2026-06-20T07:12:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4798 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:12:32 dist-test-slave-4n6h krb5kdc[20075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:12:32 dist-test-slave-4n6h krb5kdc[20075](info): set up 2 sockets
Jun 20 07:12:32 dist-test-slave-4n6h krb5kdc[20075](info): commencing operation
krb5kdc: starting...
W20260620 07:12:33.614610 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:12:33 dist-test-slave-4n6h krb5kdc[20075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:12:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:12:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:12:33.840408 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:37787
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:37787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:34.162408 20088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:34.162714 20088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:34.162875 20088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:34.170219 20088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:34.170372 20088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:34.170461 20088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:34.170562 20088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:34.170640 20088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:34.180431 20088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:37787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:37787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:34.181756 20088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:34.183017 20088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:34.194796 20096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:34.194885 20093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:34.194861 20094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:34.195670 20088 server_base.cc:1034] running on GCE node
I20260620 07:12:34.196435 20088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:34.197767 20088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:34.199007 20088 hybrid_clock.cc:648] HybridClock initialized: now 1781939554198964 us; error 46 us; skew 500 ppm
I20260620 07:12:34.199426 20088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:34 dist-test-slave-4n6h krb5kdc[20075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939554, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:34.202476 20088 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:12:34.203912 20088 webserver.cc:460] Webserver started at http://127.6.53.126:44437/ using document root <none> and password file <none>
I20260620 07:12:34.204550 20088 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:34.204643 20088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:34.204887 20088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:34.207749 20088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "891f2ed8ced046dbad06bd8c09157517"
format_stamp: "Formatted at 2026-06-20 07:12:34 on dist-test-slave-4n6h"
server_key: "afe975ada554bb6af6b41b0cc76ea900"
server_key_iv: "9aa98ab54baab2fb6380910aadad652c"
server_key_version: "encryptionkey@0"
I20260620 07:12:34.208468 20088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "891f2ed8ced046dbad06bd8c09157517"
format_stamp: "Formatted at 2026-06-20 07:12:34 on dist-test-slave-4n6h"
server_key: "afe975ada554bb6af6b41b0cc76ea900"
server_key_iv: "9aa98ab54baab2fb6380910aadad652c"
server_key_version: "encryptionkey@0"
I20260620 07:12:34.213308 20088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 07:12:34.216679 20103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:34.217934 20088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 07:12:34.218096 20088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "891f2ed8ced046dbad06bd8c09157517"
format_stamp: "Formatted at 2026-06-20 07:12:34 on dist-test-slave-4n6h"
server_key: "afe975ada554bb6af6b41b0cc76ea900"
server_key_iv: "9aa98ab54baab2fb6380910aadad652c"
server_key_version: "encryptionkey@0"
I20260620 07:12:34.218325 20088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:34.244164 20088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:34.248133 20088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:34.248459 20088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:34.267069 20088 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:37787
I20260620 07:12:34.267110 20154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:37787 every 8 connection(s)
I20260620 07:12:34.268466 20088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:12:34.272050 20155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:34.272754 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 20088
I20260620 07:12:34.272994 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:12:34.273852 6357 external_mini_cluster.cc:1428] Setting key 85c35f878f7e9140dc9e3126ed44832a
I20260620 07:12:34.286131 20155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517: Bootstrap starting.
Jun 20 07:12:34 dist-test-slave-4n6h krb5kdc[20075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:12:34.289330 20155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:34.290552 20155 log.cc:824] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:34.293407 20155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517: No bootstrap required, opened a new log
I20260620 07:12:34.297748 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:34.276004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57682 (local address 127.6.53.126:37787)
0620 07:12:34.276844 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:34.276859 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:34.276903 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:12:34.278743 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:34.278767 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:34.279172 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:34.279511 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:34.281372 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:34.282860 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:34.284271 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:34.284906 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:34.285095 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:34.289488 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:34.289497 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:34.289511 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:34.289548 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:34.292915 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:34.293694 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:34.293700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:34.293706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:34.293916 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:34.294349 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:34.294354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:34.294357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:34.294976 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:34.296291 (+ 1315us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:34.296504 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 07:12:34.296827 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":398,"threads_started":1}
I20260620 07:12:34.301416 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:12:34.301975 20155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891f2ed8ced046dbad06bd8c09157517" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 37787 } }
I20260620 07:12:34.302395 20155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:34.302490 20155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 891f2ed8ced046dbad06bd8c09157517, State: Initialized, Role: FOLLOWER
I20260620 07:12:34.303086 20155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891f2ed8ced046dbad06bd8c09157517" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 37787 } }
I20260620 07:12:34.303344 20155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:12:34.303463 20155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:12:34.303603 20155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:34.305982 20155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891f2ed8ced046dbad06bd8c09157517" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 37787 } }
I20260620 07:12:34.306466 20155 leader_election.cc:302] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 891f2ed8ced046dbad06bd8c09157517; no voters:
I20260620 07:12:34.306998 20155 leader_election.cc:288] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:12:34.307165 20160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:34.307878 20160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [term 1 LEADER]: Becoming Leader. State: Replica: 891f2ed8ced046dbad06bd8c09157517, State: Running, Role: LEADER
I20260620 07:12:34.308390 20160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891f2ed8ced046dbad06bd8c09157517" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 37787 } }
I20260620 07:12:34.308991 20155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:34.311153 20162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 891f2ed8ced046dbad06bd8c09157517. Latest consensus state: current_term: 1 leader_uuid: "891f2ed8ced046dbad06bd8c09157517" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891f2ed8ced046dbad06bd8c09157517" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 37787 } } }
I20260620 07:12:34.311257 20161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "891f2ed8ced046dbad06bd8c09157517" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891f2ed8ced046dbad06bd8c09157517" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 37787 } } }
I20260620 07:12:34.311628 20162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:34.311739 20161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:34.313864 20166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:12:34.318161 20166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:12:34.326426 20166 catalog_manager.cc:1347] Generated new cluster ID: 3e0da1e2a73e4e5fba01178684f84ec8
I20260620 07:12:34.326529 20166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:12:34.344455 20166 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:12:34.346304 20166 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:12:34.369410 20166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 891f2ed8ced046dbad06bd8c09157517: Generated new TSK 0
I20260620 07:12:34.370301 20166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:12:34.414117 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 20088
I20260620 07:12:34.415338 20150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:34.511451 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 20088
2026-06-20T07:12:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1981 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:12:34 dist-test-slave-4n6h krb5kdc[20183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:12:34 dist-test-slave-4n6h krb5kdc[20183](info): set up 2 sockets
Jun 20 07:12:34 dist-test-slave-4n6h krb5kdc[20183](info): commencing operation
krb5kdc: starting...
W20260620 07:12:35.635110 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:12:35 dist-test-slave-4n6h krb5kdc[20183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:12:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:12:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:12:35.866034 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:45559
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:45559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:36.189133 20196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:36.189414 20196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:36.189581 20196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:36.196650 20196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:36.196749 20196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:36.196835 20196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:36.196945 20196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:36.197031 20196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:36.206439 20196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:45559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:45559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:36.207715 20196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:36.208992 20196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:36.220355 20204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:36.220312 20201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:36.220381 20202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:36.221118 20196 server_base.cc:1034] running on GCE node
I20260620 07:12:36.221844 20196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:36.223276 20196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:36.224506 20196 hybrid_clock.cc:648] HybridClock initialized: now 1781939556224456 us; error 58 us; skew 500 ppm
I20260620 07:12:36.224838 20196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:36 dist-test-slave-4n6h krb5kdc[20183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939556, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:36.227905 20196 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:12:36.229341 20196 webserver.cc:460] Webserver started at http://127.6.53.126:34681/ using document root <none> and password file <none>
I20260620 07:12:36.230017 20196 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:36.230162 20196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:36.230526 20196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:36.233450 20196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "a38f6b9f4b8f4d2a85744189a541de30"
format_stamp: "Formatted at 2026-06-20 07:12:36 on dist-test-slave-4n6h"
server_key: "69d0d01c35f3730cd0133e362377b8fb"
server_key_iv: "d92a54acb3b78dd12ee19cd53b33f24d"
server_key_version: "encryptionkey@0"
I20260620 07:12:36.234174 20196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "a38f6b9f4b8f4d2a85744189a541de30"
format_stamp: "Formatted at 2026-06-20 07:12:36 on dist-test-slave-4n6h"
server_key: "69d0d01c35f3730cd0133e362377b8fb"
server_key_iv: "d92a54acb3b78dd12ee19cd53b33f24d"
server_key_version: "encryptionkey@0"
I20260620 07:12:36.239640 20196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 07:12:36.243031 20211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:36.244364 20196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:12:36.244550 20196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "a38f6b9f4b8f4d2a85744189a541de30"
format_stamp: "Formatted at 2026-06-20 07:12:36 on dist-test-slave-4n6h"
server_key: "69d0d01c35f3730cd0133e362377b8fb"
server_key_iv: "d92a54acb3b78dd12ee19cd53b33f24d"
server_key_version: "encryptionkey@0"
I20260620 07:12:36.244745 20196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:36.268827 20196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:36.272984 20196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:36.273283 20196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:36.292246 20196 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:45559
I20260620 07:12:36.292294 20262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:45559 every 8 connection(s)
I20260620 07:12:36.293545 20196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:12:36.297087 20263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:36.297945 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 20196
I20260620 07:12:36.298149 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:12:36.298862 6357 external_mini_cluster.cc:1428] Setting key 43fafa361fd95926fa39141c095d92d1
I20260620 07:12:36.311267 20263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30: Bootstrap starting.
Jun 20 07:12:36 dist-test-slave-4n6h krb5kdc[20183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:12:36.314672 20263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:36.315858 20263 log.cc:824] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:36.318727 20263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30: No bootstrap required, opened a new log
I20260620 07:12:36.323149 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:36.300803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.6.53.126:45559)
0620 07:12:36.301554 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:36.301565 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:36.301604 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:12:36.303227 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:36.303247 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:36.303597 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:36.303905 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:36.305904 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:36.307223 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:36.309778 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:36.310480 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:36.310619 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:36.315255 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:36.315264 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:36.315278 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:36.315316 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:36.318782 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:36.319677 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:36.319683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:36.319688 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:36.319867 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:36.320311 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:36.320315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:36.320317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:36.320879 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:36.321173 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:36.321613 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 07:12:36.322014 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":350,"threads_started":1}
I20260620 07:12:36.327667 20263 raft_consensus.cc:348] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } }
I20260620 07:12:36.328043 20263 raft_consensus.cc:374] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:36.328151 20263 raft_consensus.cc:729] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a38f6b9f4b8f4d2a85744189a541de30, State: Initialized, Role: FOLLOWER
I20260620 07:12:36.328639 20263 consensus_queue.cc:260] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } }
I20260620 07:12:36.328879 20263 raft_consensus.cc:388] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:12:36.329021 20263 raft_consensus.cc:482] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:12:36.329192 20263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:36.331400 20263 raft_consensus.cc:504] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } }
I20260620 07:12:36.331861 20263 leader_election.cc:302] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a38f6b9f4b8f4d2a85744189a541de30; no voters:
I20260620 07:12:36.332500 20263 leader_election.cc:288] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:12:36.332649 20268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:36.333819 20268 raft_consensus.cc:686] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 1 LEADER]: Becoming Leader. State: Replica: a38f6b9f4b8f4d2a85744189a541de30, State: Running, Role: LEADER
I20260620 07:12:36.334506 20263 sys_catalog.cc:564] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:36.334430 20268 consensus_queue.cc:237] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } }
I20260620 07:12:36.337457 20270 sys_catalog.cc:455] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a38f6b9f4b8f4d2a85744189a541de30. Latest consensus state: current_term: 1 leader_uuid: "a38f6b9f4b8f4d2a85744189a541de30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } } }
I20260620 07:12:36.337416 20269 sys_catalog.cc:455] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a38f6b9f4b8f4d2a85744189a541de30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } } }
I20260620 07:12:36.337878 20270 sys_catalog.cc:458] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:36.337883 20269 sys_catalog.cc:458] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:36.343346 20276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:12:36.347615 20276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:12:36.356562 20276 catalog_manager.cc:1347] Generated new cluster ID: 784ee847ebf84a2d94afb6ef4d71c1e6
I20260620 07:12:36.356694 20276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:12:36.364902 20276 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:12:36.366243 20276 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:12:36.379401 20276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30: Generated new TSK 0
I20260620 07:12:36.380290 20276 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:12:36.431403 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:45559
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:36.754032 20289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:36.754382 20289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:36.754560 20289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:36.761752 20289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:36.761845 20289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:36.762063 20289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:12:36.771664 20289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:45559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:36.772825 20289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:36.774063 20289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:36.786479 20294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:36.786432 20297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:36.786437 20295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:36.787640 20289 server_base.cc:1034] running on GCE node
I20260620 07:12:36.788244 20289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:36.789103 20289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:36.790359 20289 hybrid_clock.cc:648] HybridClock initialized: now 1781939556790302 us; error 56 us; skew 500 ppm
I20260620 07:12:36.790660 20289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:36 dist-test-slave-4n6h krb5kdc[20183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939556, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:36.793671 20289 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:12:36.795076 20289 webserver.cc:460] Webserver started at http://127.6.53.65:42959/ using document root <none> and password file <none>
I20260620 07:12:36.795677 20289 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:36.795770 20289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:36.796001 20289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:36.798738 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "b81a89863b5448f1a3d0c22a04cd49a7"
format_stamp: "Formatted at 2026-06-20 07:12:36 on dist-test-slave-4n6h"
server_key: "a27a9dd6565614b033d694ec0884f0e8"
server_key_iv: "8fc30981fdacab955424c16b26002952"
server_key_version: "encryptionkey@0"
I20260620 07:12:36.799410 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "b81a89863b5448f1a3d0c22a04cd49a7"
format_stamp: "Formatted at 2026-06-20 07:12:36 on dist-test-slave-4n6h"
server_key: "a27a9dd6565614b033d694ec0884f0e8"
server_key_iv: "8fc30981fdacab955424c16b26002952"
server_key_version: "encryptionkey@0"
I20260620 07:12:36.804049 20289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:12:36.807216 20304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:36.808459 20289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:12:36.808578 20289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "b81a89863b5448f1a3d0c22a04cd49a7"
format_stamp: "Formatted at 2026-06-20 07:12:36 on dist-test-slave-4n6h"
server_key: "a27a9dd6565614b033d694ec0884f0e8"
server_key_iv: "8fc30981fdacab955424c16b26002952"
server_key_version: "encryptionkey@0"
I20260620 07:12:36.808727 20289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:36.827358 20289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:36.831370 20289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:36.831652 20289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:36.832535 20289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:36.833822 20289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:36.833899 20289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:36.834069 20289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:36.834129 20289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:36.867890 20289 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:35981
I20260620 07:12:36.867928 20416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:35981 every 8 connection(s)
I20260620 07:12:36.869124 20289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:12:36.872400 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 20289
I20260620 07:12:36.872617 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:12:36.873492 6357 external_mini_cluster.cc:1428] Setting key 8850b7fc7c7c3e9a19fcbec622aedac2
Jun 20 07:12:36 dist-test-slave-4n6h krb5kdc[20183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939556, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:12:36.892398 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:36.871746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:34077 (local address 127.6.53.126:45559)
0620 07:12:36.872178 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:36.872185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:36.873364 (+ 1179us) server_negotiation.cc:407] Connection header received
0620 07:12:36.875667 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:36.875671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:36.875743 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:36.875828 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:36.877795 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:36.878650 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:36.881093 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:36.881591 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:36.881736 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:36.885655 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:36.885664 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:36.885673 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:36.885715 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:36.889184 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:36.889853 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:36.889862 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:36.889868 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:36.890019 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:36.890472 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:36.890479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:36.890485 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:36.890874 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:36.891037 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:36.891670 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 07:12:36.891911 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":230,"threads_started":1}
I20260620 07:12:36.892948 20420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:36.872302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:34077)
0620 07:12:36.873171 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:36.873227 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:36.875380 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:36.876085 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:36.876095 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:36.876831 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:36.877620 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:36.877647 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:36.878789 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:36.878797 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:36.880947 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:36.880964 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:36.881846 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:36.881856 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:36.882180 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:36.882900 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:36.882939 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:36.885434 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:36.889322 (+ 3888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:36.889329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:36.889341 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:36.889734 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:36.890125 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:36.890128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:36.890130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:36.890366 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:36.891003 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:36.891008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:36.891309 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 07:12:36.891517 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 07:12:36.891876 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":353,"threads_started":1}
I20260620 07:12:36.894498 20417 heartbeater.cc:347] Connected to a master server at 127.6.53.126:45559
I20260620 07:12:36.894805 20417 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:36.895493 20417 heartbeater.cc:511] Master 127.6.53.126:45559 requested a full tablet report, sending...
I20260620 07:12:36.897472 20228 ts_manager.cc:194] Registered new tserver with Master: b81a89863b5448f1a3d0c22a04cd49a7 (127.6.53.65:35981)
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
I20260620 07:12:36.898934 20228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:34077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:12:36.925902 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:45559
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:37.234858 20423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:37.235172 20423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:37.235318 20423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:37.242453 20423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:37.242573 20423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:37.242758 20423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:12:37.252053 20423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:45559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:37.253196 20423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:37.254415 20423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:37.266714 20428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:37.266770 20429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:37.266803 20431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:37.267987 20423 server_base.cc:1034] running on GCE node
I20260620 07:12:37.268535 20423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:37.269409 20423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:37.270633 20423 hybrid_clock.cc:648] HybridClock initialized: now 1781939557270583 us; error 55 us; skew 500 ppm
I20260620 07:12:37.270923 20423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:37 dist-test-slave-4n6h krb5kdc[20183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939557, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:37.273974 20423 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:12:37.275493 20423 webserver.cc:460] Webserver started at http://127.6.53.66:37501/ using document root <none> and password file <none>
I20260620 07:12:37.276149 20423 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:37.276288 20423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:37.276572 20423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:37.279378 20423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "393a36edb57f4b888fd840074d1a189c"
format_stamp: "Formatted at 2026-06-20 07:12:37 on dist-test-slave-4n6h"
server_key: "a7f11952423e70a3bfce454c4543702d"
server_key_iv: "cebf62dab8bc555292c1f9222003c29d"
server_key_version: "encryptionkey@0"
I20260620 07:12:37.280105 20423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "393a36edb57f4b888fd840074d1a189c"
format_stamp: "Formatted at 2026-06-20 07:12:37 on dist-test-slave-4n6h"
server_key: "a7f11952423e70a3bfce454c4543702d"
server_key_iv: "cebf62dab8bc555292c1f9222003c29d"
server_key_version: "encryptionkey@0"
I20260620 07:12:37.285410 20423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 07:12:37.288797 20438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:37.289970 20423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 07:12:37.290149 20423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "393a36edb57f4b888fd840074d1a189c"
format_stamp: "Formatted at 2026-06-20 07:12:37 on dist-test-slave-4n6h"
server_key: "a7f11952423e70a3bfce454c4543702d"
server_key_iv: "cebf62dab8bc555292c1f9222003c29d"
server_key_version: "encryptionkey@0"
I20260620 07:12:37.290392 20423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:37.314214 20423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:37.318109 20423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:37.318419 20423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:37.319344 20423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:37.320629 20423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:37.320730 20423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:37.320878 20423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:37.320955 20423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:37.353555 20423 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:42329
I20260620 07:12:37.353606 20550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:42329 every 8 connection(s)
I20260620 07:12:37.354879 20423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:12:37.356009 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 20423
I20260620 07:12:37.356201 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:12:37.356787 6357 external_mini_cluster.cc:1428] Setting key 8ddb337868145a8995e46f666f695a07
Jun 20 07:12:37 dist-test-slave-4n6h krb5kdc[20183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939557, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:12:37.375645 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:37.357686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:44349 (local address 127.6.53.126:45559)
0620 07:12:37.357866 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:37.357889 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:37.359216 (+ 1327us) server_negotiation.cc:407] Connection header received
0620 07:12:37.360989 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:37.360994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:37.361083 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:37.361199 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:37.362704 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.363437 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.365018 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.365407 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.365500 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:37.369603 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:37.369608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:37.369612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:37.369646 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:37.372414 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:37.373244 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:37.373250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:37.373254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:37.373402 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:37.373853 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:37.373858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:37.373861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:37.374158 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:37.374339 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:37.375050 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 07:12:37.375307 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:12:37.376595 20553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:37.358187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:44349)
0620 07:12:37.359067 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:37.359103 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:37.360736 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:37.361429 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:37.361437 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:37.361928 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:37.362542 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:37.362561 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.363589 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.363594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:37.364848 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:37.364864 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.365634 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.365642 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:37.366004 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:37.366744 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:37.366777 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:37.369386 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:37.372547 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:37.372555 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:37.372572 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:37.372995 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:37.373521 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:37.373526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:37.373530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:37.373729 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:37.374317 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:37.374324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:37.374680 (+ 356us) client_negotiation.cc:769] Sending connection context
0620 07:12:37.374943 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 07:12:37.375477 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":314,"threads_started":1}
I20260620 07:12:37.378288 20551 heartbeater.cc:347] Connected to a master server at 127.6.53.126:45559
I20260620 07:12:37.378598 20551 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
I20260620 07:12:37.379295 20551 heartbeater.cc:511] Master 127.6.53.126:45559 requested a full tablet report, sending...
I20260620 07:12:37.380817 20228 ts_manager.cc:194] Registered new tserver with Master: 393a36edb57f4b888fd840074d1a189c (127.6.53.66:42329)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
I20260620 07:12:37.381773 20228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:44349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:12:37.408840 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:45559
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:37.724306 20556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:37.724637 20556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:37.724828 20556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:37.732645 20556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:37.732785 20556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:37.732950 20556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:12:37.743917 20556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:45559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:37.745203 20556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:37.746543 20556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:37.759600 20564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:37.759652 20561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:37.759819 20562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:37.760692 20556 server_base.cc:1034] running on GCE node
I20260620 07:12:37.761251 20556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:37.762176 20556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:37.763437 20556 hybrid_clock.cc:648] HybridClock initialized: now 1781939557763398 us; error 37 us; skew 500 ppm
I20260620 07:12:37.763742 20556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:37 dist-test-slave-4n6h krb5kdc[20183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939557, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:37.766690 20556 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:12:37.768090 20556 webserver.cc:460] Webserver started at http://127.6.53.67:36849/ using document root <none> and password file <none>
I20260620 07:12:37.768741 20556 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:37.768886 20556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:37.769176 20556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:37.772075 20556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "c97850cb3bba4ffd817f576a179018c4"
format_stamp: "Formatted at 2026-06-20 07:12:37 on dist-test-slave-4n6h"
server_key: "e2504293a4bd50773df061c62baf5bbb"
server_key_iv: "a2113e03121d1310ffe2740336dd360b"
server_key_version: "encryptionkey@0"
I20260620 07:12:37.772835 20556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "c97850cb3bba4ffd817f576a179018c4"
format_stamp: "Formatted at 2026-06-20 07:12:37 on dist-test-slave-4n6h"
server_key: "e2504293a4bd50773df061c62baf5bbb"
server_key_iv: "a2113e03121d1310ffe2740336dd360b"
server_key_version: "encryptionkey@0"
I20260620 07:12:37.777685 20556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:12:37.780936 20571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:37.782228 20556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:37.782471 20556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "c97850cb3bba4ffd817f576a179018c4"
format_stamp: "Formatted at 2026-06-20 07:12:37 on dist-test-slave-4n6h"
server_key: "e2504293a4bd50773df061c62baf5bbb"
server_key_iv: "a2113e03121d1310ffe2740336dd360b"
server_key_version: "encryptionkey@0"
I20260620 07:12:37.782665 20556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:37.800318 20556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:37.804060 20556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:37.804368 20556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:37.805258 20556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:37.806596 20556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:37.806700 20556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:37.806852 20556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:37.806924 20556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:37.845378 20556 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:42371
I20260620 07:12:37.845430 20683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:42371 every 8 connection(s)
I20260620 07:12:37.847008 20556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:12:37.850948 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 20556
I20260620 07:12:37.851131 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:12:37.851825 6357 external_mini_cluster.cc:1428] Setting key c87a68b98e977a5d17da4bec01857191
Jun 20 07:12:37 dist-test-slave-4n6h krb5kdc[20183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939557, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:12:37.872428 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:37.850365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:44951 (local address 127.6.53.126:45559)
0620 07:12:37.850566 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:37.850574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:37.852142 (+ 1568us) server_negotiation.cc:407] Connection header received
0620 07:12:37.854936 (+ 2794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:37.854943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:37.855035 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:37.855123 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:37.857160 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.858400 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.860041 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.860803 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.860949 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:37.865449 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:37.865455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:37.865460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:37.865523 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:37.869309 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:37.870003 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:37.870009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:37.870012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:37.870150 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:37.870642 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:37.870647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:37.870650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:37.871061 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:37.871211 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:37.871851 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 07:12:37.872096 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:12:37.873175 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:37.851059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:44951)
0620 07:12:37.851972 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:37.852019 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:37.854628 (+ 2609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:37.855410 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:37.855422 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:37.856229 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:37.856992 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:37.857017 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.858561 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.858566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:37.859877 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:37.859889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.861049 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.861058 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:37.861414 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:37.862357 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:37.862386 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:37.865179 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:37.869446 (+ 4267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:37.869454 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:37.869466 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:37.869883 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:37.870317 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:37.870324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:37.870329 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:37.870529 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:37.871193 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:37.871207 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:37.871496 (+ 289us) client_negotiation.cc:769] Sending connection context
0620 07:12:37.871719 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 07:12:37.872089 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":348,"threads_started":1}
I20260620 07:12:37.874528 20684 heartbeater.cc:347] Connected to a master server at 127.6.53.126:45559
I20260620 07:12:37.874825 20684 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:37.875541 20684 heartbeater.cc:511] Master 127.6.53.126:45559 requested a full tablet report, sending...
I20260620 07:12:37.876842 20228 ts_manager.cc:194] Registered new tserver with Master: c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371)
I20260620 07:12:37.877611 20228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:44951
I20260620 07:12:37.880378 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:12:37.896924 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:37.884602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.6.53.126:45559)
0620 07:12:37.884785 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:37.884791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:37.885173 (+ 382us) server_negotiation.cc:407] Connection header received
0620 07:12:37.885376 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:37.885381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:37.885446 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:37.885548 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:37.887034 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.887734 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.889030 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.889424 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.889564 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:37.891837 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:37.891841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:37.891845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:37.891874 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:37.894431 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:37.895053 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:37.895059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:37.895063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:37.895195 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:37.895585 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:37.895592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:37.895597 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:37.895890 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:37.896132 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:37.896410 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 07:12:37.896621 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:12:37.901886 20417 heartbeater.cc:503] Master 127.6.53.126:45559 was elected leader, sending a full tablet report...
I20260620 07:12:37.900457 20228 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:12:37.907770 20228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:12:37.931087 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:37.918977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.6.53.65:35981)
0620 07:12:37.920436 (+ 1459us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:37.920444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:37.921565 (+ 1121us) server_negotiation.cc:407] Connection header received
0620 07:12:37.922374 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:37.922383 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:37.922612 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:37.922773 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:37.923859 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.925540 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.928455 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.929517 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.929660 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:37.929905 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:37.930559 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 07:12:37.930746 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1312,"thread_start_us":1161,"threads_started":1}
I20260620 07:12:37.931454 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:37.921570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.6.53.66:42329)
0620 07:12:37.923840 (+ 2270us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:37.923850 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:37.924298 (+ 448us) server_negotiation.cc:407] Connection header received
0620 07:12:37.924503 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:37.924512 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:37.924699 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:37.924881 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:37.925411 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.926776 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.928840 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.930216 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.930360 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:37.930538 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:37.931018 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 07:12:37.931166 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2066,"thread_start_us":1959,"threads_started":1}
I20260620 07:12:37.931563 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:37.921033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:42329 (local address 127.0.0.1:53850)
0620 07:12:37.924175 (+ 3142us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:37.924192 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:37.924321 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:37.924865 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:37.924869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:37.924894 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:37.925240 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:37.925249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.926930 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.926935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:37.928601 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:37.928615 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.930698 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.930713 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:37.930828 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:37.930863 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:12:37.930964 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 07:12:37.931138 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3017,"mutex_wait_us":3,"thread_start_us":803,"threads_started":1}
I20260620 07:12:37.931588 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:37.919070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:42371 (local address 127.0.0.1:33952)
0620 07:12:37.922209 (+ 3139us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:37.922232 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:37.922400 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:37.923929 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:37.923933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:37.923959 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:37.924445 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:37.924457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.926418 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.926424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:37.928635 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:37.928645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.930733 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.930739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:37.930925 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:37.930958 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:12:37.931043 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:12:37.931177 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2969,"mutex_wait_us":24,"thread_start_us":1982,"threads_started":1}
I20260620 07:12:37.932188 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:37.918922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:35981 (local address 127.0.0.1:54462)
0620 07:12:37.921434 (+ 2512us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:37.921473 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:37.921710 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:37.923109 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:37.923119 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:37.923158 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:37.923639 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:37.923651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.925699 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.925704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:37.928282 (+ 2578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:37.928298 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.930048 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.930056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:37.930297 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:37.930359 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 07:12:37.931676 (+ 1317us) client_negotiation.cc:240] Negotiation successful
0620 07:12:37.931795 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2259,"mutex_wait_us":81,"thread_start_us":4964,"threads_started":1}
I20260620 07:12:37.932202 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:37.919679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.6.53.67:42371)
0620 07:12:37.920903 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:37.920914 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:37.922572 (+ 1658us) server_negotiation.cc:407] Connection header received
0620 07:12:37.922699 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:37.922709 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:37.922953 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:37.923142 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:37.924618 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.926203 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.928782 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:37.930219 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:37.930360 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:37.930567 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:37.931711 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0620 07:12:37.931860 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1055,"thread_start_us":900,"threads_started":1}
I20260620 07:12:37.935518 20352 tablet_service.cc:1467] Processing CreateTablet for tablet 8f0daf56fb0c4716b242ec30042d5ccd (DEFAULT_TABLE table=test-table [id=55781fc0ed0a4eb9b296c977a2a7b91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:37.935521 20619 tablet_service.cc:1467] Processing CreateTablet for tablet 8f0daf56fb0c4716b242ec30042d5ccd (DEFAULT_TABLE table=test-table [id=55781fc0ed0a4eb9b296c977a2a7b91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:37.935443 20486 tablet_service.cc:1467] Processing CreateTablet for tablet 8f0daf56fb0c4716b242ec30042d5ccd (DEFAULT_TABLE table=test-table [id=55781fc0ed0a4eb9b296c977a2a7b91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:37.936683 20352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0daf56fb0c4716b242ec30042d5ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:37.936745 20486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0daf56fb0c4716b242ec30042d5ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:37.936736 20619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0daf56fb0c4716b242ec30042d5ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:37.948660 20702 tablet_bootstrap.cc:492] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7: Bootstrap starting.
I20260620 07:12:37.950682 20704 tablet_bootstrap.cc:492] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c: Bootstrap starting.
I20260620 07:12:37.951403 20703 tablet_bootstrap.cc:492] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4: Bootstrap starting.
I20260620 07:12:37.952258 20702 tablet_bootstrap.cc:654] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:37.953518 20702 log.cc:824] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:37.954820 20703 tablet_bootstrap.cc:654] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:37.956027 20703 log.cc:824] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:37.956097 20702 tablet_bootstrap.cc:492] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7: No bootstrap required, opened a new log
I20260620 07:12:37.956382 20702 ts_tablet_manager.cc:1397] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 07:12:37.957656 20704 tablet_bootstrap.cc:654] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:37.958746 20704 log.cc:824] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:37.960232 20703 tablet_bootstrap.cc:492] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4: No bootstrap required, opened a new log
I20260620 07:12:37.960477 20703 ts_tablet_manager.cc:1397] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 07:12:37.961532 20704 tablet_bootstrap.cc:492] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c: No bootstrap required, opened a new log
I20260620 07:12:37.961772 20704 ts_tablet_manager.cc:1397] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 07:12:37.964280 20702 raft_consensus.cc:348] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:37.964617 20702 raft_consensus.cc:374] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:37.964748 20702 raft_consensus.cc:729] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81a89863b5448f1a3d0c22a04cd49a7, State: Initialized, Role: FOLLOWER
I20260620 07:12:37.965317 20702 consensus_queue.cc:260] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:37.967058 20702 ts_tablet_manager.cc:1428] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20260620 07:12:37.970935 20703 raft_consensus.cc:348] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:37.971381 20703 raft_consensus.cc:374] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:37.971482 20703 raft_consensus.cc:729] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97850cb3bba4ffd817f576a179018c4, State: Initialized, Role: FOLLOWER
I20260620 07:12:37.972092 20703 consensus_queue.cc:260] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:37.973105 20684 heartbeater.cc:503] Master 127.6.53.126:45559 was elected leader, sending a full tablet report...
I20260620 07:12:37.973800 20703 ts_tablet_manager.cc:1428] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 07:12:37.973553 20704 raft_consensus.cc:348] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:37.973958 20704 raft_consensus.cc:374] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:37.974058 20704 raft_consensus.cc:729] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 393a36edb57f4b888fd840074d1a189c, State: Initialized, Role: FOLLOWER
I20260620 07:12:37.974712 20704 consensus_queue.cc:260] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:37.975726 20551 heartbeater.cc:503] Master 127.6.53.126:45559 was elected leader, sending a full tablet report...
I20260620 07:12:37.976240 20704 ts_tablet_manager.cc:1428] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260620 07:12:38.073361 20710 raft_consensus.cc:482] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:38.073782 20710 raft_consensus.cc:504] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:38.075536 20710 leader_election.cc:288] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b81a89863b5448f1a3d0c22a04cd49a7 (127.6.53.65:35981), c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371)
I20260620 07:12:38.085470 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:38.076077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:45047 (local address 127.6.53.65:35981)
0620 07:12:38.076269 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:38.076275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:38.076984 (+ 709us) server_negotiation.cc:407] Connection header received
0620 07:12:38.077179 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:38.077184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:38.077265 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:38.077365 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:38.078362 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:38.079473 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:38.082913 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:38.084161 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:38.084307 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:38.084406 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:38.085083 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 07:12:38.085234 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:12:38.085508 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:38.076078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:52585 (local address 127.6.53.67:42371)
0620 07:12:38.076269 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:38.076275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:38.076984 (+ 709us) server_negotiation.cc:407] Connection header received
0620 07:12:38.077179 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:38.077184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:38.077265 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:38.077365 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:38.078362 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:38.079473 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:38.082925 (+ 3452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:38.084157 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:38.084299 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:38.084406 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:38.085083 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 07:12:38.085218 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:12:38.085865 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:38.075816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:42371 (local address 127.6.53.66:52585)
0620 07:12:38.076849 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:38.076869 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:38.077033 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:38.077451 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:38.077457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:38.077496 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:38.078200 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:38.078212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:38.079626 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:38.079632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:38.082727 (+ 3095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:38.082737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:38.084480 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:38.084487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:38.084820 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:38.084935 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 07:12:38.085047 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 07:12:38.085286 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":341,"spinlock_wait_cycles":16000,"thread_start_us":640,"threads_started":1}
I20260620 07:12:38.085959 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:38.075869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:35981 (local address 127.6.53.66:45047)
0620 07:12:38.076854 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:38.076874 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:38.077033 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:38.077451 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:38.077457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:38.077500 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:38.078126 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:38.078152 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:38.079628 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:38.079632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:38.082719 (+ 3087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:38.082737 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:38.084484 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:38.084488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:38.084793 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:38.084935 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 07:12:38.085047 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 07:12:38.085245 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":344,"spinlock_wait_cycles":29440,"thread_start_us":614,"threads_started":1}
I20260620 07:12:38.087147 20372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0daf56fb0c4716b242ec30042d5ccd" candidate_uuid: "393a36edb57f4b888fd840074d1a189c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" is_pre_election: true
I20260620 07:12:38.087240 20639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0daf56fb0c4716b242ec30042d5ccd" candidate_uuid: "393a36edb57f4b888fd840074d1a189c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97850cb3bba4ffd817f576a179018c4" is_pre_election: true
I20260620 07:12:38.087684 20372 raft_consensus.cc:2454] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 393a36edb57f4b888fd840074d1a189c in term 0.
I20260620 07:12:38.087810 20639 raft_consensus.cc:2454] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 393a36edb57f4b888fd840074d1a189c in term 0.
I20260620 07:12:38.088419 20439 leader_election.cc:302] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 393a36edb57f4b888fd840074d1a189c, b81a89863b5448f1a3d0c22a04cd49a7; no voters:
I20260620 07:12:38.088815 20710 raft_consensus.cc:2781] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:12:38.088963 20710 raft_consensus.cc:482] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:12:38.089069 20710 raft_consensus.cc:3037] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:38.091398 20710 raft_consensus.cc:504] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:38.091996 20710 leader_election.cc:288] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [CANDIDATE]: Term 1 election: Requested vote from peers b81a89863b5448f1a3d0c22a04cd49a7 (127.6.53.65:35981), c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371)
I20260620 07:12:38.092420 20372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0daf56fb0c4716b242ec30042d5ccd" candidate_uuid: "393a36edb57f4b888fd840074d1a189c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81a89863b5448f1a3d0c22a04cd49a7"
I20260620 07:12:38.092512 20639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0daf56fb0c4716b242ec30042d5ccd" candidate_uuid: "393a36edb57f4b888fd840074d1a189c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97850cb3bba4ffd817f576a179018c4"
I20260620 07:12:38.092614 20372 raft_consensus.cc:3037] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:38.092757 20639 raft_consensus.cc:3037] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:38.095134 20372 raft_consensus.cc:2454] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 393a36edb57f4b888fd840074d1a189c in term 1.
I20260620 07:12:38.095209 20639 raft_consensus.cc:2454] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 393a36edb57f4b888fd840074d1a189c in term 1.
I20260620 07:12:38.095510 20439 leader_election.cc:302] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 393a36edb57f4b888fd840074d1a189c, b81a89863b5448f1a3d0c22a04cd49a7; no voters:
I20260620 07:12:38.095836 20710 raft_consensus.cc:2781] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:38.096244 20710 raft_consensus.cc:686] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [term 1 LEADER]: Becoming Leader. State: Replica: 393a36edb57f4b888fd840074d1a189c, State: Running, Role: LEADER
I20260620 07:12:38.096684 20710 consensus_queue.cc:237] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
W20260620 07:12:38.098819 20685 tablet.cc:2369] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:12:38.101106 20227 catalog_manager.cc:5495] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c reported cstate change: term changed from 0 to 1, leader changed from <none> to 393a36edb57f4b888fd840074d1a189c (127.6.53.66). New cstate: current_term: 1 leader_uuid: "393a36edb57f4b888fd840074d1a189c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } health_report { overall_health: HEALTHY } } }
W20260620 07:12:38.106596 20552 tablet.cc:2369] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:12:38.117782 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:38.112538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.6.53.66:42329)
0620 07:12:38.112753 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:38.112757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:38.112864 (+ 107us) server_negotiation.cc:407] Connection header received
0620 07:12:38.113123 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:38.113128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:38.113206 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:38.113333 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:12:38.113646 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:38.114475 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:38.115813 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:38.116250 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:38.116344 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:38.116761 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:12:38.117236 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:12:38.117350 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:38.117410 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 07:12:38.117471 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 07:12:38.121485 20418 tablet.cc:2369] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:12:38.125533 20372 raft_consensus.cc:1261] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Refusing update from remote peer 393a36edb57f4b888fd840074d1a189c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:38.125536 20639 raft_consensus.cc:1261] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4 [term 1 FOLLOWER]: Refusing update from remote peer 393a36edb57f4b888fd840074d1a189c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:38.126585 20710 consensus_queue.cc:1035] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:38.126889 20713 consensus_queue.cc:1035] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:38.137137 20716 mvcc.cc:205] Tried to move back new op lower bound from 7298824430072274944 to 7298824429967843328. Current Snapshot: MvccSnapshot[applied={T|T < 7298824430072274944}]
I20260620 07:12:38.138020 20717 mvcc.cc:205] Tried to move back new op lower bound from 7298824430072274944 to 7298824429967843328. Current Snapshot: MvccSnapshot[applied={T|T < 7298824430072274944}]
I20260620 07:12:38.151237 20227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:38.151501 20227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:38.155038 20227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30: Sending DeleteTablet for 3 replicas of tablet 8f0daf56fb0c4716b242ec30042d5ccd
I20260620 07:12:38.156148 20352 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f0daf56fb0c4716b242ec30042d5ccd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:38 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:54462
I20260620 07:12:38.156153 20619 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f0daf56fb0c4716b242ec30042d5ccd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:38 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:33952
I20260620 07:12:38.156684 20486 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f0daf56fb0c4716b242ec30042d5ccd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:38 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:53850
I20260620 07:12:38.157447 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 20289
I20260620 07:12:38.171025 20728 tablet_replica.cc:330] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c: stopping tablet replica
I20260620 07:12:38.171159 20726 tablet_replica.cc:330] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4: stopping tablet replica
I20260620 07:12:38.171651 20726 raft_consensus.cc:2229] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:38.171895 20728 raft_consensus.cc:2229] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:12:38.172151 20726 raft_consensus.cc:2258] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:38.172914 20727 tablet_replica.cc:330] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7: stopping tablet replica
I20260620 07:12:38.173015 20728 raft_consensus.cc:2258] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:38.173283 20727 raft_consensus.cc:2229] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:38.173846 20727 raft_consensus.cc:2258] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:38.175984 20728 ts_tablet_manager.cc:1905] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:38.175984 20727 ts_tablet_manager.cc:1905] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:38.177615 20726 ts_tablet_manager.cc:1905] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 07:12:38 dist-test-slave-4n6h krb5kdc[20183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:12:38.183391 20727 ts_tablet_manager.cc:1918] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:38.183432 20728 ts_tablet_manager.cc:1918] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:38.183619 20728 log.cc:1196] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/8f0daf56fb0c4716b242ec30042d5ccd
I20260620 07:12:38.183650 20727 log.cc:1196] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/8f0daf56fb0c4716b242ec30042d5ccd
I20260620 07:12:38.184322 20727 ts_tablet_manager.cc:1939] T 8f0daf56fb0c4716b242ec30042d5ccd P b81a89863b5448f1a3d0c22a04cd49a7: Deleting consensus metadata
I20260620 07:12:38.184535 20728 ts_tablet_manager.cc:1939] T 8f0daf56fb0c4716b242ec30042d5ccd P 393a36edb57f4b888fd840074d1a189c: Deleting consensus metadata
I20260620 07:12:38.186197 20213 catalog_manager.cc:4841] TS 393a36edb57f4b888fd840074d1a189c (127.6.53.66:42329): tablet 8f0daf56fb0c4716b242ec30042d5ccd (table test-table [id=55781fc0ed0a4eb9b296c977a2a7b91e]) successfully deleted
I20260620 07:12:38.186393 20212 catalog_manager.cc:4841] TS b81a89863b5448f1a3d0c22a04cd49a7 (127.6.53.65:35981): tablet 8f0daf56fb0c4716b242ec30042d5ccd (table test-table [id=55781fc0ed0a4eb9b296c977a2a7b91e]) successfully deleted
I20260620 07:12:38.190165 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:38.158555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54468 (local address 127.6.53.65:35981)
0620 07:12:38.158738 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:38.158747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:38.166474 (+ 7727us) server_negotiation.cc:407] Connection header received
0620 07:12:38.167618 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:38.167627 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:38.167770 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:38.167880 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:38.173439 (+ 5559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:38.174758 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:38.176823 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:38.177468 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:38.177569 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:38.183454 (+ 5885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:38.183467 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:38.183487 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:38.183532 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:38.187375 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:38.188110 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:38.188121 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:38.188129 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:38.188248 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:38.188626 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:38.188631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:38.188634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:38.188975 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:38.189272 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:38.189579 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 07:12:38.189795 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 07:12:38.190816 20412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:38.190811 20726 ts_tablet_manager.cc:1918] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:38.190961 20726 log.cc:1196] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/8f0daf56fb0c4716b242ec30042d5ccd
I20260620 07:12:38.191540 20726 ts_tablet_manager.cc:1939] T 8f0daf56fb0c4716b242ec30042d5ccd P c97850cb3bba4ffd817f576a179018c4: Deleting consensus metadata
I20260620 07:12:38.193570 20215 catalog_manager.cc:4841] TS c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371): tablet 8f0daf56fb0c4716b242ec30042d5ccd (table test-table [id=55781fc0ed0a4eb9b296c977a2a7b91e]) successfully deleted
I20260620 07:12:38.281642 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 20289
I20260620 07:12:38.305583 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 20423
Jun 20 07:12:38 dist-test-slave-4n6h krb5kdc[20183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:12:38.322568 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:38.306396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.6.53.66:42329)
0620 07:12:38.306577 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:38.306583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:38.306651 (+ 68us) server_negotiation.cc:407] Connection header received
0620 07:12:38.306847 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:38.306852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:38.306930 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:38.307054 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:38.308335 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:38.309546 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:38.310970 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:38.311645 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:38.311769 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:38.315889 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:38.315906 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:38.315931 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:38.315981 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:38.319801 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:38.320458 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:38.320469 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:38.320473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:38.320649 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:38.320989 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:38.320996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:38.321004 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:38.321440 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:38.321764 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:38.321927 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 07:12:38.322136 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:12:38.323351 20546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:38.439005 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 20423
I20260620 07:12:38.457633 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 20556
Jun 20 07:12:38 dist-test-slave-4n6h krb5kdc[20183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:12:38.472668 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:38.458482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.6.53.67:42371)
0620 07:12:38.458653 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:38.458657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:38.458744 (+ 87us) server_negotiation.cc:407] Connection header received
0620 07:12:38.458905 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:38.458908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:38.458973 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:38.459050 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:38.460513 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:38.461220 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:38.462591 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:38.462955 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:38.463052 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:38.467567 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:38.467580 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:38.467595 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:38.467630 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:38.470499 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:38.471073 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:38.471080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:38.471083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:38.471228 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:38.471543 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:38.471546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:38.471548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:38.471858 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:38.472075 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:38.472277 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 07:12:38.472438 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:12:38.473308 20679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:38.563405 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 20556
I20260620 07:12:38.579996 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 20196
I20260620 07:12:38.581161 20258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:38.693224 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 20196
I20260620 07:12:38.716286 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:45559
--webserver_interface=127.6.53.126
--webserver_port=34681
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:45559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:39.033136 20734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:39.033474 20734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:39.033638 20734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:39.041179 20734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:39.041355 20734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:39.041491 20734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:39.041608 20734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:39.041708 20734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:39.052466 20734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:45559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:45559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=34681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:39.053758 20734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:39.055086 20734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:39.076735 20740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:39.076793 20743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:39.076792 20741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:39.077807 20734 server_base.cc:1034] running on GCE node
I20260620 07:12:39.078703 20734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:39.080197 20734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:39.081395 20734 hybrid_clock.cc:648] HybridClock initialized: now 1781939559081353 us; error 33 us; skew 500 ppm
I20260620 07:12:39.081781 20734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:39 dist-test-slave-4n6h krb5kdc[20183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939559, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:39.085474 20734 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:12:39.087039 20734 webserver.cc:460] Webserver started at http://127.6.53.126:34681/ using document root <none> and password file <none>
I20260620 07:12:39.087731 20734 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:39.087855 20734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:39.093600 20734 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 07:12:39.096499 20750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:39.097822 20734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:39.098001 20734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "a38f6b9f4b8f4d2a85744189a541de30"
format_stamp: "Formatted at 2026-06-20 07:12:36 on dist-test-slave-4n6h"
server_key: "69d0d01c35f3730cd0133e362377b8fb"
server_key_iv: "d92a54acb3b78dd12ee19cd53b33f24d"
server_key_version: "encryptionkey@0"
I20260620 07:12:39.099084 20734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:39.116899 20734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:39.121256 20734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:39.121532 20734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:39.140199 20734 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:45559
I20260620 07:12:39.140218 20801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:45559 every 8 connection(s)
I20260620 07:12:39.141707 20734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:12:39.147083 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 20734
I20260620 07:12:39.147871 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:35981
--local_ip_for_outbound_sockets=127.6.53.65
--tserver_master_addrs=127.6.53.126:45559
--webserver_port=42959
--webserver_interface=127.6.53.65
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:12:39.148839 20802 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:12:39.152148 20802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30: Bootstrap starting.
I20260620 07:12:39.179795 20802 log.cc:824] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:39.195051 20802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:12:39.195696 20802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30: Bootstrap complete.
I20260620 07:12:39.209836 20802 raft_consensus.cc:348] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } }
I20260620 07:12:39.211118 20802 raft_consensus.cc:729] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a38f6b9f4b8f4d2a85744189a541de30, State: Initialized, Role: FOLLOWER
I20260620 07:12:39.211879 20802 consensus_queue.cc:260] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } }
I20260620 07:12:39.212230 20802 raft_consensus.cc:388] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:12:39.212401 20802 raft_consensus.cc:482] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:12:39.212621 20802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:12:39.219054 20802 raft_consensus.cc:504] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } }
I20260620 07:12:39.219659 20802 leader_election.cc:302] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a38f6b9f4b8f4d2a85744189a541de30; no voters:
I20260620 07:12:39.220283 20802 leader_election.cc:288] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:12:39.220439 20806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:12:39.221647 20806 raft_consensus.cc:686] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 2 LEADER]: Becoming Leader. State: Replica: a38f6b9f4b8f4d2a85744189a541de30, State: Running, Role: LEADER
I20260620 07:12:39.222224 20806 consensus_queue.cc:237] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } }
I20260620 07:12:39.222551 20802 sys_catalog.cc:564] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:39.224799 20808 sys_catalog.cc:455] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a38f6b9f4b8f4d2a85744189a541de30. Latest consensus state: current_term: 2 leader_uuid: "a38f6b9f4b8f4d2a85744189a541de30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } } }
I20260620 07:12:39.224794 20807 sys_catalog.cc:455] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a38f6b9f4b8f4d2a85744189a541de30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } } }
I20260620 07:12:39.225232 20807 sys_catalog.cc:458] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:39.225230 20808 sys_catalog.cc:458] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:39.237591 20812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:12:39.245453 20812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:12:39.246913 20812 catalog_manager.cc:1259] Loaded cluster ID: 784ee847ebf84a2d94afb6ef4d71c1e6
I20260620 07:12:39.247059 20812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:12:39.250167 20812 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 07:12:39.268514 20812 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 07:12:39.277616 20812 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:12:39.279284 20812 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30: Loaded TSK: 0
I20260620 07:12:39.280726 20812 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:12:39.516296 20804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:39.516603 20804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:39.516762 20804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:39.524209 20804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:39.524338 20804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:39.524497 20804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:12:39.534055 20804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:35981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=42959
--tserver_master_addrs=127.6.53.126:45559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:39.535337 20804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:39.536604 20804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:39.550745 20829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:39.550961 20832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:39.551011 20830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:39.552831 20804 server_base.cc:1034] running on GCE node
I20260620 07:12:39.553440 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:39.554370 20804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:39.555600 20804 hybrid_clock.cc:648] HybridClock initialized: now 1781939559555524 us; error 56 us; skew 500 ppm
I20260620 07:12:39.555934 20804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:39 dist-test-slave-4n6h krb5kdc[20183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939559, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:39.559125 20804 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:12:39.560758 20804 webserver.cc:460] Webserver started at http://127.6.53.65:42959/ using document root <none> and password file <none>
I20260620 07:12:39.561568 20804 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:39.561698 20804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:39.568171 20804 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:12:39.571348 20839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:39.572952 20804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:12:39.573098 20804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "b81a89863b5448f1a3d0c22a04cd49a7"
format_stamp: "Formatted at 2026-06-20 07:12:36 on dist-test-slave-4n6h"
server_key: "a27a9dd6565614b033d694ec0884f0e8"
server_key_iv: "8fc30981fdacab955424c16b26002952"
server_key_version: "encryptionkey@0"
I20260620 07:12:39.574011 20804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:39.596863 20804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:39.600800 20804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:39.601087 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:39.602082 20804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:39.603477 20804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:39.603559 20804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:39.603686 20804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:39.603766 20804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:39.635216 20804 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:35981
I20260620 07:12:39.635294 20951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:35981 every 8 connection(s)
I20260620 07:12:39.636557 20804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:12:39.644001 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 20804
I20260620 07:12:39.644585 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:42329
--local_ip_for_outbound_sockets=127.6.53.66
--tserver_master_addrs=127.6.53.126:45559
--webserver_port=37501
--webserver_interface=127.6.53.66
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:12:39 dist-test-slave-4n6h krb5kdc[20183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939559, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:12:39.658973 20955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:39.639308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:45793)
0620 07:12:39.640061 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:39.640096 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:39.641721 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:39.642951 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:39.642958 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:39.643421 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:39.643922 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:39.643937 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:39.645679 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:39.645684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:39.646942 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:39.646953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:39.647938 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:39.647945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:39.648217 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:39.648850 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:39.648871 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:39.651133 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:39.655350 (+ 4217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:39.655356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:39.655365 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:39.655756 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:39.656225 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:39.656228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:39.656231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:39.656384 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:39.657196 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:39.657201 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:39.657441 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 07:12:39.657639 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 07:12:39.657952 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":323,"threads_started":1}
I20260620 07:12:39.659474 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:39.639212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:45793 (local address 127.6.53.126:45559)
0620 07:12:39.639865 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:39.639880 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:39.640233 (+ 353us) server_negotiation.cc:407] Connection header received
0620 07:12:39.642169 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:39.642180 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:39.642526 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:39.642783 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:39.644159 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:39.645521 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:39.647086 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:39.647686 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:39.647900 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:39.651393 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:39.651400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:39.651412 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:39.651450 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:39.655218 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:39.655883 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:39.655894 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:39.655903 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:39.656100 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:39.656478 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:39.656486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:39.656492 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:39.657078 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:39.657268 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:39.657869 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 07:12:39.658340 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":235,"threads_started":1}
I20260620 07:12:39.661938 20952 heartbeater.cc:347] Connected to a master server at 127.6.53.126:45559
I20260620 07:12:39.662281 20952 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:39.663251 20952 heartbeater.cc:511] Master 127.6.53.126:45559 requested a full tablet report, sending...
I20260620 07:12:39.665546 20767 ts_manager.cc:194] Registered new tserver with Master: b81a89863b5448f1a3d0c22a04cd49a7 (127.6.53.65:35981)
I20260620 07:12:39.667049 20767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:45793
W20260620 07:12:39.966522 20956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:39.966817 20956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:39.966969 20956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:39.973986 20956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:39.974120 20956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:39.974315 20956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:12:39.983861 20956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:42329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=37501
--tserver_master_addrs=127.6.53.126:45559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:39.985019 20956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:39.986285 20956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:39.998843 20961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:39.998865 20962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:39.998831 20964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:40.000070 20956 server_base.cc:1034] running on GCE node
I20260620 07:12:40.000639 20956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:40.001559 20956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:40.002786 20956 hybrid_clock.cc:648] HybridClock initialized: now 1781939560002731 us; error 43 us; skew 500 ppm
I20260620 07:12:40.003150 20956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:40 dist-test-slave-4n6h krb5kdc[20183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939560, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:40.006330 20956 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:12:40.007687 20956 webserver.cc:460] Webserver started at http://127.6.53.66:37501/ using document root <none> and password file <none>
I20260620 07:12:40.008412 20956 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:40.008561 20956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:40.013706 20956 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:40.016242 20971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:40.017433 20956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:12:40.017596 20956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "393a36edb57f4b888fd840074d1a189c"
format_stamp: "Formatted at 2026-06-20 07:12:37 on dist-test-slave-4n6h"
server_key: "a7f11952423e70a3bfce454c4543702d"
server_key_iv: "cebf62dab8bc555292c1f9222003c29d"
server_key_version: "encryptionkey@0"
I20260620 07:12:40.018687 20956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:40.055222 20956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:40.059160 20956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:40.059434 20956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:40.060374 20956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:40.061982 20956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:40.062108 20956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:40.062247 20956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:40.062374 20956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 07:12:40.099681 20956 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:42329
I20260620 07:12:40.099730 21083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:42329 every 8 connection(s)
I20260620 07:12:40.101253 20956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:12:40.109790 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 20956
I20260620 07:12:40.110608 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:42371
--local_ip_for_outbound_sockets=127.6.53.67
--tserver_master_addrs=127.6.53.126:45559
--webserver_port=36849
--webserver_interface=127.6.53.67
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:12:40 dist-test-slave-4n6h krb5kdc[20183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939560, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:12:40.120270 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.103908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:46857 (local address 127.6.53.126:45559)
0620 07:12:40.104085 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:40.104089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:40.105299 (+ 1210us) server_negotiation.cc:407] Connection header received
0620 07:12:40.107287 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:40.107293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:40.107375 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:40.107477 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:40.108868 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.109559 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.111201 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.111608 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.111738 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.114938 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:40.114943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:40.114946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:40.114977 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:40.117347 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:40.118061 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:40.118066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:40.118074 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:40.118207 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:40.118655 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:40.118660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:40.118663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:40.118920 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:40.119066 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:40.119709 (+ 643us) server_negotiation.cc:299] Negotiation successful
0620 07:12:40.119932 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:12:40.121115 21086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.104468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:46857)
0620 07:12:40.105160 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:40.105197 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:40.107021 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:40.107703 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:40.107711 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:40.108197 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:40.108706 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:40.108721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.109720 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.109724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:40.111033 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:40.111045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.111834 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.111841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:40.112061 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.112743 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:40.112767 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:40.114746 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:40.117491 (+ 2745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:40.117498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:40.117512 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:40.117925 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:40.118352 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:40.118357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:40.118361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:40.118540 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:40.119050 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:40.119057 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:40.119352 (+ 295us) client_negotiation.cc:769] Sending connection context
0620 07:12:40.119598 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 07:12:40.120082 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":279,"threads_started":1}
I20260620 07:12:40.122501 21084 heartbeater.cc:347] Connected to a master server at 127.6.53.126:45559
I20260620 07:12:40.122823 21084 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:40.123566 21084 heartbeater.cc:511] Master 127.6.53.126:45559 requested a full tablet report, sending...
I20260620 07:12:40.124962 20767 ts_manager.cc:194] Registered new tserver with Master: 393a36edb57f4b888fd840074d1a189c (127.6.53.66:42329)
I20260620 07:12:40.125857 20767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:46857
W20260620 07:12:40.432330 21087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:40.432600 21087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:40.432691 21087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:40.439791 21087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:40.439894 21087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:40.440011 21087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:12:40.449456 21087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:42371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=36849
--tserver_master_addrs=127.6.53.126:45559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:40.450657 21087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:40.451854 21087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:40.464277 21092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:40.464195 21093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:40.464253 21095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:40.465099 21087 server_base.cc:1034] running on GCE node
I20260620 07:12:40.465651 21087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:40.466497 21087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:40.467715 21087 hybrid_clock.cc:648] HybridClock initialized: now 1781939560467649 us; error 30 us; skew 500 ppm
I20260620 07:12:40.468016 21087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:40 dist-test-slave-4n6h krb5kdc[20183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939560, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:40.471045 21087 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:12:40.472398 21087 webserver.cc:460] Webserver started at http://127.6.53.67:36849/ using document root <none> and password file <none>
I20260620 07:12:40.473032 21087 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:40.473182 21087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:40.478214 21087 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:12:40.480794 21102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:40.482066 21087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:12:40.482312 21087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "c97850cb3bba4ffd817f576a179018c4"
format_stamp: "Formatted at 2026-06-20 07:12:37 on dist-test-slave-4n6h"
server_key: "e2504293a4bd50773df061c62baf5bbb"
server_key_iv: "a2113e03121d1310ffe2740336dd360b"
server_key_version: "encryptionkey@0"
I20260620 07:12:40.483314 21087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:40.506635 21087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:40.510908 21087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:40.511178 21087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:40.512132 21087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:40.513399 21087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:40.513518 21087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:40.513654 21087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:40.513744 21087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:40.545511 21087 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:42371
I20260620 07:12:40.545547 21214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:42371 every 8 connection(s)
I20260620 07:12:40.546990 21087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:12:40.552984 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 21087
Jun 20 07:12:40 dist-test-slave-4n6h krb5kdc[20183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939560, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:12:40.568776 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.549451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:42275 (local address 127.6.53.126:45559)
0620 07:12:40.549650 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:40.549657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:40.550800 (+ 1143us) server_negotiation.cc:407] Connection header received
0620 07:12:40.552528 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:40.552534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:40.552637 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:40.552757 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:40.554479 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.555635 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.557299 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.557735 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.557871 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.561661 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:40.561670 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:40.561676 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:40.561724 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:40.565109 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:40.565974 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:40.565978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:40.565980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:40.566178 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:40.566705 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:40.566713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:40.566720 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:40.567152 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:40.567316 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:40.568143 (+ 827us) server_negotiation.cc:299] Negotiation successful
0620 07:12:40.568349 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:12:40.569396 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.553822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.6.53.126:45559)
0620 07:12:40.554324 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:40.554329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:40.554847 (+ 518us) server_negotiation.cc:407] Connection header received
0620 07:12:40.555019 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:40.555022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:40.555078 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:40.555159 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:40.556798 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.557723 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.559166 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.559814 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.559988 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.563295 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:40.563300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:40.563303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:40.563343 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:40.566468 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:40.567290 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:40.567295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:40.567298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:40.567429 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:40.567829 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:40.567835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:40.567838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:40.568218 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:40.568473 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:40.568863 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 07:12:40.569071 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":253,"threads_started":1}
I20260620 07:12:40.569733 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.549939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:42275)
0620 07:12:40.550647 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:40.550685 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:40.552276 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:40.552977 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:40.552984 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:40.553537 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:40.554267 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:40.554293 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.555781 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.555785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:40.557143 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:40.557157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.557951 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.557955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:40.558133 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.558827 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:40.558848 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:40.561431 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:40.565247 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:40.565258 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:40.565275 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:40.565839 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:40.566330 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:40.566335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:40.566339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:40.566572 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:40.567295 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:40.567302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:40.567672 (+ 370us) client_negotiation.cc:769] Sending connection context
0620 07:12:40.567984 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 07:12:40.568483 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":311,"threads_started":1}
I20260620 07:12:40.571182 21215 heartbeater.cc:347] Connected to a master server at 127.6.53.126:45559
I20260620 07:12:40.571466 21215 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:40.572185 21215 heartbeater.cc:511] Master 127.6.53.126:45559 requested a full tablet report, sending...
I20260620 07:12:40.573447 20767 ts_manager.cc:194] Registered new tserver with Master: c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371)
I20260620 07:12:40.574331 20767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:42275
I20260620 07:12:40.584281 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:12:40.591177 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.585238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.6.53.126:45559)
0620 07:12:40.585414 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:40.585418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:40.585882 (+ 464us) server_negotiation.cc:407] Connection header received
0620 07:12:40.586095 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:40.586098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:40.586314 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:40.586471 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:12:40.586875 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.587628 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.589042 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.589465 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.589598 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.590085 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:12:40.590624 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:12:40.590788 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:40.590840 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 07:12:40.590918 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:12:40.592064 20767 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:12:40.599422 20767 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:12:40.642310 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.613008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53878 (local address 127.6.53.66:42329)
0620 07:12:40.623631 (+ 10623us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:40.623644 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:40.632581 (+ 8937us) server_negotiation.cc:407] Connection header received
0620 07:12:40.632695 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:40.632702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:40.632917 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:40.633115 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:40.634011 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.635509 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.639025 (+ 3516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.640783 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.640893 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.641082 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:40.641802 (+ 720us) server_negotiation.cc:299] Negotiation successful
0620 07:12:40.641984 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10409,"thread_start_us":8439,"threads_started":1}
I20260620 07:12:40.642477 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.610159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:42371 (local address 127.0.0.1:33970)
0620 07:12:40.625950 (+ 15791us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:40.626014 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:40.626202 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:40.632746 (+ 6544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:40.632752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:40.632803 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:40.633326 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:40.633341 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.635514 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.635519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:40.638797 (+ 3278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:40.638815 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.641318 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.641323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:40.641514 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.641584 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 07:12:40.641726 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 07:12:40.641934 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15475,"mutex_wait_us":170,"spinlock_wait_cycles":5888,"thread_start_us":15238,"threads_started":1}
I20260620 07:12:40.643177 21226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.610437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33970 (local address 127.6.53.67:42371)
0620 07:12:40.632109 (+ 21672us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:40.632120 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:40.632168 (+ 48us) server_negotiation.cc:407] Connection header received
0620 07:12:40.632266 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:40.632276 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:40.632506 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:40.632687 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:40.633574 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.635352 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.639082 (+ 3730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.640783 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.640907 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.641118 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:40.642649 (+ 1531us) server_negotiation.cc:299] Negotiation successful
0620 07:12:40.642821 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21476,"thread_start_us":21309,"threads_started":1}
I20260620 07:12:40.643307 21228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.610619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54470 (local address 127.6.53.65:35981)
0620 07:12:40.631660 (+ 21041us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:40.631684 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:40.631722 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:12:40.631855 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:40.631868 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:40.632182 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:40.632407 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:40.633936 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.635495 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.638882 (+ 3387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.641394 (+ 2512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.641511 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.641745 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:40.642904 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0620 07:12:40.643027 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20824,"thread_start_us":20680,"threads_started":1}
I20260620 07:12:40.642794 21229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.610555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:35981 (local address 127.0.0.1:54470)
0620 07:12:40.630861 (+ 20306us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:40.630887 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:40.631035 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:40.632743 (+ 1708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:40.632750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:40.632789 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:40.633718 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:40.633734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.635660 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.635669 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:40.638669 (+ 3000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:40.638693 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.641937 (+ 3244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.641944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:40.642112 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.642145 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:12:40.642232 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 07:12:40.642400 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20127,"mutex_wait_us":136,"spinlock_wait_cycles":1920,"thread_start_us":21555,"threads_started":1}
I20260620 07:12:40.643425 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.612410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:42329 (local address 127.0.0.1:53878)
0620 07:12:40.632423 (+ 20013us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:40.632440 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:40.632573 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:40.633114 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:40.633118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:40.633150 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:40.633792 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:40.633808 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.635690 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.635698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:40.638826 (+ 3128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:40.638836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.641307 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.641313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:40.641521 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.641584 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 07:12:40.642894 (+ 1310us) client_negotiation.cc:240] Negotiation successful
0620 07:12:40.643045 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19884,"mutex_wait_us":143,"thread_start_us":18093,"threads_started":1}
I20260620 07:12:40.646687 21150 tablet_service.cc:1467] Processing CreateTablet for tablet bf04470b13b7463cbdaa4884808da1f8 (DEFAULT_TABLE table=test-table [id=e09232ec04704a23bf4f7a5734fa4a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:40.646970 21019 tablet_service.cc:1467] Processing CreateTablet for tablet bf04470b13b7463cbdaa4884808da1f8 (DEFAULT_TABLE table=test-table [id=e09232ec04704a23bf4f7a5734fa4a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:40.648033 21150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf04470b13b7463cbdaa4884808da1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:40.648137 21019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf04470b13b7463cbdaa4884808da1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:40.649708 20887 tablet_service.cc:1467] Processing CreateTablet for tablet bf04470b13b7463cbdaa4884808da1f8 (DEFAULT_TABLE table=test-table [id=e09232ec04704a23bf4f7a5734fa4a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:40.651011 20887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf04470b13b7463cbdaa4884808da1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:40.661293 21231 tablet_bootstrap.cc:492] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c: Bootstrap starting.
I20260620 07:12:40.664486 21231 tablet_bootstrap.cc:654] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:40.665336 21232 tablet_bootstrap.cc:492] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4: Bootstrap starting.
I20260620 07:12:40.665902 21231 log.cc:824] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:40.668044 21233 tablet_bootstrap.cc:492] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7: Bootstrap starting.
I20260620 07:12:40.668953 21232 tablet_bootstrap.cc:654] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:40.669209 21231 tablet_bootstrap.cc:492] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c: No bootstrap required, opened a new log
I20260620 07:12:40.669479 21231 ts_tablet_manager.cc:1397] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 07:12:40.670138 21232 log.cc:824] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:40.670717 20952 heartbeater.cc:503] Master 127.6.53.126:45559 was elected leader, sending a full tablet report...
I20260620 07:12:40.670931 21233 tablet_bootstrap.cc:654] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:40.672091 21233 log.cc:824] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:40.674894 21233 tablet_bootstrap.cc:492] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7: No bootstrap required, opened a new log
I20260620 07:12:40.675284 21233 ts_tablet_manager.cc:1397] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 07:12:40.677234 21232 tablet_bootstrap.cc:492] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4: No bootstrap required, opened a new log
I20260620 07:12:40.677578 21232 ts_tablet_manager.cc:1397] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260620 07:12:40.681406 21231 raft_consensus.cc:348] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:40.681843 21231 raft_consensus.cc:374] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:40.682008 21231 raft_consensus.cc:729] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 393a36edb57f4b888fd840074d1a189c, State: Initialized, Role: FOLLOWER
I20260620 07:12:40.682631 21231 consensus_queue.cc:260] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:40.682775 21233 raft_consensus.cc:348] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:40.683176 21233 raft_consensus.cc:374] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:40.683274 21233 raft_consensus.cc:729] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81a89863b5448f1a3d0c22a04cd49a7, State: Initialized, Role: FOLLOWER
I20260620 07:12:40.683786 21233 consensus_queue.cc:260] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:40.685438 21233 ts_tablet_manager.cc:1428] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260620 07:12:40.686302 21084 heartbeater.cc:503] Master 127.6.53.126:45559 was elected leader, sending a full tablet report...
I20260620 07:12:40.686949 21231 ts_tablet_manager.cc:1428] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c: Time spent starting tablet: real 0.017s user 0.008s sys 0.008s
I20260620 07:12:40.689531 21232 raft_consensus.cc:348] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:40.689924 21232 raft_consensus.cc:374] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:40.690023 21232 raft_consensus.cc:729] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97850cb3bba4ffd817f576a179018c4, State: Initialized, Role: FOLLOWER
I20260620 07:12:40.690641 21232 consensus_queue.cc:260] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:40.691751 21215 heartbeater.cc:503] Master 127.6.53.126:45559 was elected leader, sending a full tablet report...
I20260620 07:12:40.692266 21232 ts_tablet_manager.cc:1428] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260620 07:12:40.699669 21238 raft_consensus.cc:482] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:40.699889 21238 raft_consensus.cc:504] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:40.701318 21238 leader_election.cc:288] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371), 393a36edb57f4b888fd840074d1a189c (127.6.53.66:42329)
I20260620 07:12:40.707775 21226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.701686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:33105 (local address 127.6.53.67:42371)
0620 07:12:40.701905 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:40.701912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:40.702605 (+ 693us) server_negotiation.cc:407] Connection header received
0620 07:12:40.702778 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:40.702788 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:40.702842 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:40.702933 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:40.703538 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.704264 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.706015 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.706852 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.706991 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.707097 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:40.707393 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 07:12:40.707511 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260620 07:12:40.707932 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.701706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:55677 (local address 127.6.53.66:42329)
0620 07:12:40.701907 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:40.701913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:40.702685 (+ 772us) server_negotiation.cc:407] Connection header received
0620 07:12:40.702877 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:40.702882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:40.702947 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:40.703038 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:40.703776 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.704464 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.706161 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.706988 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.707132 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.707232 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:40.707534 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 07:12:40.707666 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:12:40.707978 21240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.701540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:42371 (local address 127.6.53.65:33105)
0620 07:12:40.702505 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:40.702528 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:40.702645 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:40.702947 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:40.702959 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:40.702985 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:40.703402 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:40.703413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.704381 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.704388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:40.705869 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:40.705881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.707062 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.707078 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:40.707229 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.707251 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:12:40.707368 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 07:12:40.707491 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":2,"thread_start_us":395,"threads_started":1}
I20260620 07:12:40.708094 21241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.701540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:42329 (local address 127.6.53.65:55677)
0620 07:12:40.702598 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:40.702617 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:40.702744 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:40.703055 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:40.703059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:40.703115 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:40.703616 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:40.703628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.704614 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.704620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:40.706015 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:40.706026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.707215 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.707221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:40.707386 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.707411 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 07:12:40.707509 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 07:12:40.707630 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":225,"spinlock_wait_cycles":18176,"thread_start_us":477,"threads_started":1}
I20260620 07:12:40.708891 21170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf04470b13b7463cbdaa4884808da1f8" candidate_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97850cb3bba4ffd817f576a179018c4" is_pre_election: true
I20260620 07:12:40.708912 21039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf04470b13b7463cbdaa4884808da1f8" candidate_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "393a36edb57f4b888fd840074d1a189c" is_pre_election: true
I20260620 07:12:40.709331 21039 raft_consensus.cc:2454] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81a89863b5448f1a3d0c22a04cd49a7 in term 0.
I20260620 07:12:40.709331 21170 raft_consensus.cc:2454] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81a89863b5448f1a3d0c22a04cd49a7 in term 0.
I20260620 07:12:40.709983 20841 leader_election.cc:302] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 393a36edb57f4b888fd840074d1a189c, b81a89863b5448f1a3d0c22a04cd49a7, c97850cb3bba4ffd817f576a179018c4; no voters:
I20260620 07:12:40.710657 21238 raft_consensus.cc:2781] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:12:40.710811 21238 raft_consensus.cc:482] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:12:40.710916 21238 raft_consensus.cc:3037] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:40.713039 21238 raft_consensus.cc:504] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:40.713666 21238 leader_election.cc:288] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 1 election: Requested vote from peers c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371), 393a36edb57f4b888fd840074d1a189c (127.6.53.66:42329)
I20260620 07:12:40.714018 21170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf04470b13b7463cbdaa4884808da1f8" candidate_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97850cb3bba4ffd817f576a179018c4"
I20260620 07:12:40.714221 21170 raft_consensus.cc:3037] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:40.714169 21039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf04470b13b7463cbdaa4884808da1f8" candidate_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "393a36edb57f4b888fd840074d1a189c"
I20260620 07:12:40.714439 21039 raft_consensus.cc:3037] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:40.716468 21170 raft_consensus.cc:2454] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81a89863b5448f1a3d0c22a04cd49a7 in term 1.
I20260620 07:12:40.716645 21039 raft_consensus.cc:2454] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81a89863b5448f1a3d0c22a04cd49a7 in term 1.
I20260620 07:12:40.716861 20843 leader_election.cc:302] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81a89863b5448f1a3d0c22a04cd49a7, c97850cb3bba4ffd817f576a179018c4; no voters:
I20260620 07:12:40.717170 21238 raft_consensus.cc:2781] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:40.717582 21238 raft_consensus.cc:686] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 LEADER]: Becoming Leader. State: Replica: b81a89863b5448f1a3d0c22a04cd49a7, State: Running, Role: LEADER
I20260620 07:12:40.718014 21238 consensus_queue.cc:237] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:40.722635 20766 catalog_manager.cc:5495] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b81a89863b5448f1a3d0c22a04cd49a7 (127.6.53.65). New cstate: current_term: 1 leader_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } health_report { overall_health: UNKNOWN } } }
I20260620 07:12:40.746378 21228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.741403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54476 (local address 127.6.53.65:35981)
0620 07:12:40.741570 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:40.741574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:40.741592 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:12:40.741771 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:40.741775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:40.741845 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:40.741960 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:12:40.742293 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.742983 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.744355 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.744712 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.744800 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.745296 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:12:40.745742 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:12:40.745876 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:40.746000 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 07:12:40.746106 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:12:40.754624 21039 raft_consensus.cc:1261] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c [term 1 FOLLOWER]: Refusing update from remote peer b81a89863b5448f1a3d0c22a04cd49a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:40.754631 21170 raft_consensus.cc:1261] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4 [term 1 FOLLOWER]: Refusing update from remote peer b81a89863b5448f1a3d0c22a04cd49a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:40.755532 21242 consensus_queue.cc:1035] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:40.755875 21238 consensus_queue.cc:1035] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:40.774449 21245 mvcc.cc:205] Tried to move back new op lower bound from 7298824440841900032 to 7298824440704561152. Current Snapshot: MvccSnapshot[applied={T|T < 7298824440841900032 or (T in {7298824440841900032})}]
I20260620 07:12:40.780153 20766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:40.780438 20766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:40.783998 20766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30: Sending DeleteTablet for 3 replicas of tablet bf04470b13b7463cbdaa4884808da1f8
I20260620 07:12:40.785030 21150 tablet_service.cc:1514] Processing DeleteTablet for tablet bf04470b13b7463cbdaa4884808da1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:40 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:33970
I20260620 07:12:40.785065 20887 tablet_service.cc:1514] Processing DeleteTablet for tablet bf04470b13b7463cbdaa4884808da1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:40 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:54470
I20260620 07:12:40.785369 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 20804
I20260620 07:12:40.785450 21019 tablet_service.cc:1514] Processing DeleteTablet for tablet bf04470b13b7463cbdaa4884808da1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:40 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:53878
I20260620 07:12:40.787984 21257 tablet_replica.cc:330] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c: stopping tablet replica
I20260620 07:12:40.788532 21257 raft_consensus.cc:2229] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:40.789119 21257 raft_consensus.cc:2258] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:40.792989 21257 ts_tablet_manager.cc:1905] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:40.790846 21256 tablet_replica.cc:330] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4: stopping tablet replica
I20260620 07:12:40.793983 21256 raft_consensus.cc:2229] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:40.799003 21256 raft_consensus.cc:2258] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:40.800339 21255 tablet_replica.cc:330] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7: stopping tablet replica
I20260620 07:12:40.800778 21255 raft_consensus.cc:2229] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:12:40.801354 21255 raft_consensus.cc:2258] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:40.803975 21256 ts_tablet_manager.cc:1905] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:40.805701 21228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.786582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.6.53.65:35981)
0620 07:12:40.786793 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:40.786801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:40.786823 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:12:40.786910 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:40.786918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:40.787010 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:40.787181 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:40.789262 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.790790 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.793382 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.794110 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.794230 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.798528 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:40.798542 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:40.798558 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:40.798595 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:40.802747 (+ 4152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:40.803425 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:40.803433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:40.803437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:40.803584 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:40.804010 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:40.804015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:40.804019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:40.804434 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:40.804774 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:40.805042 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 07:12:40.805300 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:12:40.806434 20947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:40.807600 21257 ts_tablet_manager.cc:1918] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:40.807772 21257 log.cc:1196] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/bf04470b13b7463cbdaa4884808da1f8
I20260620 07:12:40.808259 21257 ts_tablet_manager.cc:1939] T bf04470b13b7463cbdaa4884808da1f8 P 393a36edb57f4b888fd840074d1a189c: Deleting consensus metadata
I20260620 07:12:40.809738 20752 catalog_manager.cc:4841] TS 393a36edb57f4b888fd840074d1a189c (127.6.53.66:42329): tablet bf04470b13b7463cbdaa4884808da1f8 (table test-table [id=e09232ec04704a23bf4f7a5734fa4a52]) successfully deleted
I20260620 07:12:40.813040 21256 ts_tablet_manager.cc:1918] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:40.813381 21256 log.cc:1196] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/bf04470b13b7463cbdaa4884808da1f8
I20260620 07:12:40.814038 21256 ts_tablet_manager.cc:1939] T bf04470b13b7463cbdaa4884808da1f8 P c97850cb3bba4ffd817f576a179018c4: Deleting consensus metadata
I20260620 07:12:40.815475 20754 catalog_manager.cc:4841] TS c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371): tablet bf04470b13b7463cbdaa4884808da1f8 (table test-table [id=e09232ec04704a23bf4f7a5734fa4a52]) successfully deleted
I20260620 07:12:40.904035 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 20804
I20260620 07:12:40.922289 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 20956
W20260620 07:12:40.922289 20751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.53.65:35981 (error 108)
I20260620 07:12:40.924090 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.923666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:35981 (local address 127.0.0.1:54486)
0620 07:12:40.923839 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:40.923952 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 07:12:40.924422 20751 catalog_manager.cc:4584] TS b81a89863b5448f1a3d0c22a04cd49a7 (127.6.53.65:35981): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf04470b13b7463cbdaa4884808da1f8: Network error: Client connection negotiation failed: client connection to 127.6.53.65:35981: connect: Connection refused (error 111)
I20260620 07:12:40.935318 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.923021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.6.53.66:42329)
0620 07:12:40.923223 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:40.923230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:40.923406 (+ 176us) server_negotiation.cc:407] Connection header received
0620 07:12:40.923591 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:40.923597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:40.923675 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:40.923789 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:40.925271 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.925998 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.927367 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:40.927746 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:40.927848 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:40.930230 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:40.930242 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:40.930301 (+ 59us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:40.930331 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:40.932973 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:40.933530 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:40.933536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:40.933538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:40.933649 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:40.933942 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:40.933945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:40.933947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:40.934394 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:40.934630 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:40.934811 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 07:12:40.934960 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:12:40.935899 21079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:40.977990 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.977440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:35981 (local address 127.0.0.1:54494)
0620 07:12:40.977693 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:40.977796 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 07:12:40.979125 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:40.978702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:35981 (local address 127.0.0.1:54498)
0620 07:12:40.978894 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:40.978989 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 07:12:41.024180 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 20956
I20260620 07:12:41.039839 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:41.039347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:35981 (local address 127.0.0.1:54510)
0620 07:12:41.039562 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:41.039652 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 07:12:41.040869 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:41.040579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:35981 (local address 127.0.0.1:54516)
0620 07:12:41.040698 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:41.040762 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 07:12:41.041548 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 21087
I20260620 07:12:41.053845 21226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:41.042308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33984 (local address 127.6.53.67:42371)
0620 07:12:41.042459 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:41.042463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:41.042481 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:12:41.042585 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:41.042589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:41.042669 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:41.042752 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:41.044039 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:41.044743 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:41.045937 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:41.046354 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:41.046443 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:41.048855 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:41.048869 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:41.048886 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:41.048918 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:41.051556 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:41.052166 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:41.052172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:41.052174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:41.052293 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:41.052600 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:41.052603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:41.052605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:41.052968 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:41.053196 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:41.053440 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 07:12:41.053590 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:12:41.054546 21210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:41.121820 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:41.121221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:35981 (local address 127.0.0.1:54530)
0620 07:12:41.121558 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:41.121653 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260620 07:12:41.122973 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:41.122552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:35981 (local address 127.0.0.1:54544)
0620 07:12:41.122786 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:41.122872 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260620 07:12:41.143791 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 21087
I20260620 07:12:41.160440 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 20734
I20260620 07:12:41.161542 20797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:41.273313 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 20734
I20260620 07:12:41.295390 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:45559
--webserver_interface=127.6.53.126
--webserver_port=34681
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:45559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:41.617712 21263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:41.618016 21263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:41.618191 21263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:41.625478 21263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:41.625579 21263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:41.625684 21263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:41.625783 21263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:41.625855 21263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:41.635872 21263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:45559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:45559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=34681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:41.637197 21263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:41.638477 21263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:41.649804 21271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:41.649853 21269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:41.650095 21268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:41.650803 21263 server_base.cc:1034] running on GCE node
I20260620 07:12:41.651532 21263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:41.653211 21263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:41.654475 21263 hybrid_clock.cc:648] HybridClock initialized: now 1781939561654364 us; error 90 us; skew 500 ppm
I20260620 07:12:41.654815 21263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:41 dist-test-slave-4n6h krb5kdc[20183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939561, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:41.658849 21263 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:12:41.660609 21263 webserver.cc:460] Webserver started at http://127.6.53.126:34681/ using document root <none> and password file <none>
I20260620 07:12:41.661549 21263 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:41.661705 21263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:41.667219 21263 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:12:41.669845 21278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:41.671200 21263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:12:41.671334 21263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "a38f6b9f4b8f4d2a85744189a541de30"
format_stamp: "Formatted at 2026-06-20 07:12:36 on dist-test-slave-4n6h"
server_key: "69d0d01c35f3730cd0133e362377b8fb"
server_key_iv: "d92a54acb3b78dd12ee19cd53b33f24d"
server_key_version: "encryptionkey@0"
I20260620 07:12:41.672273 21263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:41.688158 21263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:41.692217 21263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:41.692427 21263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:41.709741 21263 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:45559
I20260620 07:12:41.709772 21329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:45559 every 8 connection(s)
I20260620 07:12:41.711181 21263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:12:41.716591 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 21263
I20260620 07:12:41.717415 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:35981
--local_ip_for_outbound_sockets=127.6.53.65
--tserver_master_addrs=127.6.53.126:45559
--webserver_port=42959
--webserver_interface=127.6.53.65
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:12:41.717905 21330 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:12:41.721053 21330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30: Bootstrap starting.
I20260620 07:12:41.749686 21330 log.cc:824] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:41.766314 21330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:12:41.766868 21330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30: Bootstrap complete.
I20260620 07:12:41.776330 21330 raft_consensus.cc:348] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } }
I20260620 07:12:41.777240 21330 raft_consensus.cc:729] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a38f6b9f4b8f4d2a85744189a541de30, State: Initialized, Role: FOLLOWER
I20260620 07:12:41.777817 21330 consensus_queue.cc:260] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } }
I20260620 07:12:41.778044 21330 raft_consensus.cc:388] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:12:41.778170 21330 raft_consensus.cc:482] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:12:41.778396 21330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 2 FOLLOWER]: Advancing to term 3
I20260620 07:12:41.782445 21330 raft_consensus.cc:504] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } }
I20260620 07:12:41.782963 21330 leader_election.cc:302] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a38f6b9f4b8f4d2a85744189a541de30; no voters:
I20260620 07:12:41.783602 21330 leader_election.cc:288] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 07:12:41.783798 21334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 07:12:41.784989 21334 raft_consensus.cc:686] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [term 3 LEADER]: Becoming Leader. State: Replica: a38f6b9f4b8f4d2a85744189a541de30, State: Running, Role: LEADER
I20260620 07:12:41.785668 21330 sys_catalog.cc:564] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:41.785526 21334 consensus_queue.cc:237] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } }
I20260620 07:12:41.788300 21335 sys_catalog.cc:455] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a38f6b9f4b8f4d2a85744189a541de30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } } }
I20260620 07:12:41.788268 21336 sys_catalog.cc:455] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a38f6b9f4b8f4d2a85744189a541de30. Latest consensus state: current_term: 3 leader_uuid: "a38f6b9f4b8f4d2a85744189a541de30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38f6b9f4b8f4d2a85744189a541de30" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45559 } } }
I20260620 07:12:41.788719 21335 sys_catalog.cc:458] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:41.788766 21336 sys_catalog.cc:458] T 00000000000000000000000000000000 P a38f6b9f4b8f4d2a85744189a541de30 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:41.800406 21339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:12:41.811107 21339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:12:41.813066 21339 catalog_manager.cc:1259] Loaded cluster ID: 784ee847ebf84a2d94afb6ef4d71c1e6
I20260620 07:12:41.813200 21339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 07:12:41.815768 21339 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7efec7f7f9ad google::LogMessage::Fail() at ??:0
@ 0x7efec7f8288a google::LogMessage::SendToLog() at ??:0
@ 0x7efec7f7f4de google::LogMessage::Flush() at ??:0
@ 0x7efec7f80a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efedd9b2512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7efedd9af976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7efedda37610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7efedda37439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7efecabcb345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7efecabce91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7efecabce799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7efecaba9eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7efecde836db start_thread at ??:0
@ 0x7efec6e1961f clone at ??:0
W20260620 07:12:42.115088 21332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:42.115386 21332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:42.115589 21332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:42.123555 21332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:42.123683 21332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:42.123867 21332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:12:42.133530 21332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:35981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=42959
--tserver_master_addrs=127.6.53.126:45559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:42.134828 21332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:42.136178 21332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:42.148607 21370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:42.148696 21373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:42.148849 21371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:42.150238 21332 server_base.cc:1034] running on GCE node
I20260620 07:12:42.150831 21332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:42.151726 21332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:42.152902 21332 hybrid_clock.cc:648] HybridClock initialized: now 1781939562152863 us; error 29 us; skew 500 ppm
I20260620 07:12:42.153260 21332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:42 dist-test-slave-4n6h krb5kdc[20183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:42.156395 21332 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:12:42.157770 21332 webserver.cc:460] Webserver started at http://127.6.53.65:42959/ using document root <none> and password file <none>
I20260620 07:12:42.158529 21332 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:42.158663 21332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:42.163823 21332 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 07:12:42.166730 21380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:42.167963 21332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 07:12:42.168159 21332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "b81a89863b5448f1a3d0c22a04cd49a7"
format_stamp: "Formatted at 2026-06-20 07:12:36 on dist-test-slave-4n6h"
server_key: "a27a9dd6565614b033d694ec0884f0e8"
server_key_iv: "8fc30981fdacab955424c16b26002952"
server_key_version: "encryptionkey@0"
I20260620 07:12:42.169088 21332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:42.187638 21332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:42.191852 21332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:42.192131 21332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:42.193044 21332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:42.194746 21387 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 07:12:42.200146 21332 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 07:12:42.200278 21332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 07:12:42.200412 21332 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 07:12:42.203583 21332 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 07:12:42.203677 21332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:42.204000 21387 tablet_bootstrap.cc:492] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7: Bootstrap starting.
I20260620 07:12:42.213774 21387 log.cc:824] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:42.228700 21387 tablet_bootstrap.cc:492] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:12:42.229554 21387 tablet_bootstrap.cc:492] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7: Bootstrap complete.
I20260620 07:12:42.230180 21387 ts_tablet_manager.cc:1397] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.013s
I20260620 07:12:42.240583 21387 raft_consensus.cc:348] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:42.241796 21387 raft_consensus.cc:729] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81a89863b5448f1a3d0c22a04cd49a7, State: Initialized, Role: FOLLOWER
I20260620 07:12:42.242401 21332 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:35981
I20260620 07:12:42.242439 21494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:35981 every 8 connection(s)
I20260620 07:12:42.242630 21387 consensus_queue.cc:260] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:42.243854 21332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:12:42.244096 21387 ts_tablet_manager.cc:1428] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260620 07:12:42.244663 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 21332
I20260620 07:12:42.245474 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:42329
--local_ip_for_outbound_sockets=127.6.53.66
--tserver_master_addrs=127.6.53.126:45559
--webserver_port=37501
--webserver_interface=127.6.53.66
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:12:42.248761 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:42.246902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:37903)
0620 07:12:42.247713 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:42.247924 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":369,"threads_started":1}
W20260620 07:12:42.249462 21496 heartbeater.cc:644] Failed to heartbeat to 127.6.53.126:45559 (0 consecutive failures): Network error: Failed to ping master at 127.6.53.126:45559: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
I20260620 07:12:42.250335 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:42.249951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:59117)
0620 07:12:42.250094 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:42.250184 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 07:12:42.251245 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:42.250904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:51773)
0620 07:12:42.251046 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:42.251125 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 07:12:42.251526 21496 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:12:42.572285 21499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:42.572539 21499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:42.572721 21499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:42.579856 21499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:42.579982 21499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:42.580166 21499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:12:42.589870 21499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:42329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=37501
--tserver_master_addrs=127.6.53.126:45559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:42.591115 21499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:42.592379 21499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:42.604969 21504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:42.605060 21505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:42.605175 21507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:42.606078 21499 server_base.cc:1034] running on GCE node
I20260620 07:12:42.606724 21499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:42.607698 21499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:42.608922 21499 hybrid_clock.cc:648] HybridClock initialized: now 1781939562608889 us; error 49 us; skew 500 ppm
I20260620 07:12:42.609236 21499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:42 dist-test-slave-4n6h krb5kdc[20183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:42.612264 21499 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:12:42.613607 21499 webserver.cc:460] Webserver started at http://127.6.53.66:37501/ using document root <none> and password file <none>
I20260620 07:12:42.614305 21499 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:42.614437 21499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:42.619496 21499 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 07:12:42.622103 21514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:42.623410 21499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:12:42.623570 21499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "393a36edb57f4b888fd840074d1a189c"
format_stamp: "Formatted at 2026-06-20 07:12:37 on dist-test-slave-4n6h"
server_key: "a7f11952423e70a3bfce454c4543702d"
server_key_iv: "cebf62dab8bc555292c1f9222003c29d"
server_key_version: "encryptionkey@0"
I20260620 07:12:42.624545 21499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:42.642100 21499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:42.646149 21499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:42.646426 21499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:42.647373 21499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:42.648615 21499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:42.648739 21499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:42.648864 21499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:42.648948 21499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:42.681377 21499 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:42329
I20260620 07:12:42.681455 21626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:42329 every 8 connection(s)
I20260620 07:12:42.682965 21499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:12:42.687525 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:42.685756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:48189)
0620 07:12:42.686513 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:42.686704 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":297,"threads_started":1}
W20260620 07:12:42.688159 21627 heartbeater.cc:644] Failed to heartbeat to 127.6.53.126:45559 (0 consecutive failures): Network error: Failed to ping master at 127.6.53.126:45559: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
I20260620 07:12:42.688936 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 21499
I20260620 07:12:42.689019 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:42.688612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:59765)
0620 07:12:42.688789 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:42.688875 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 07:12:42.689640 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:42371
--local_ip_for_outbound_sockets=127.6.53.67
--tserver_master_addrs=127.6.53.126:45559
--webserver_port=36849
--webserver_interface=127.6.53.67
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:12:42.689997 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:42.689577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:50403)
0620 07:12:42.689747 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:42.689851 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 07:12:42.690327 21627 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:12:43.002017 21630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:43.002331 21630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:43.002512 21630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:43.009686 21630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:43.009811 21630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:43.009992 21630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:12:43.019433 21630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:42371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=36849
--tserver_master_addrs=127.6.53.126:45559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:12:43.020598 21630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:43.021849 21630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:43.034770 21636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:43.034977 21635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:43.035216 21638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:43.035629 21630 server_base.cc:1034] running on GCE node
I20260620 07:12:43.036144 21630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:43.036938 21630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:43.038193 21630 hybrid_clock.cc:648] HybridClock initialized: now 1781939563038134 us; error 44 us; skew 500 ppm
I20260620 07:12:43.038530 21630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:12:43 dist-test-slave-4n6h krb5kdc[20183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939563, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:43.041532 21630 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:12:43.043001 21630 webserver.cc:460] Webserver started at http://127.6.53.67:36849/ using document root <none> and password file <none>
I20260620 07:12:43.043673 21630 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:43.043809 21630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:43.048769 21630 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 07:12:43.051285 21645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:43.052492 21630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:12:43.052666 21630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "c97850cb3bba4ffd817f576a179018c4"
format_stamp: "Formatted at 2026-06-20 07:12:37 on dist-test-slave-4n6h"
server_key: "e2504293a4bd50773df061c62baf5bbb"
server_key_iv: "a2113e03121d1310ffe2740336dd360b"
server_key_version: "encryptionkey@0"
I20260620 07:12:43.053586 21630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:43.065891 21630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:43.069834 21630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:43.070097 21630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:43.071002 21630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:43.072237 21630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:43.072342 21630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:43.072454 21630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:43.072531 21630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:43.104678 21630 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:42371
I20260620 07:12:43.104760 21757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:42371 every 8 connection(s)
I20260620 07:12:43.106091 21630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:12:43.110217 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 21630
I20260620 07:12:43.110888 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:43.108975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:51659)
0620 07:12:43.109715 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:43.109943 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":281,"threads_started":1}
W20260620 07:12:43.112066 21758 heartbeater.cc:644] Failed to heartbeat to 127.6.53.126:45559 (0 consecutive failures): Network error: Failed to ping master at 127.6.53.126:45559: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
I20260620 07:12:43.112828 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:43.112517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:34399)
0620 07:12:43.112665 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:43.112731 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 07:12:43.113601 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:43.113328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:59863)
0620 07:12:43.113438 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:43.113503 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 07:12:43.113891 21758 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:12:43.252624 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:43.251958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:35419)
0620 07:12:43.252330 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:43.252417 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":177,"threads_started":1}
I20260620 07:12:43.433264 21763 raft_consensus.cc:482] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:43.433494 21763 raft_consensus.cc:504] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:43.435132 21763 leader_election.cc:288] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371), 393a36edb57f4b888fd840074d1a189c (127.6.53.66:42329)
Jun 20 07:12:43 dist-test-slave-4n6h krb5kdc[20183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
Jun 20 07:12:43 dist-test-slave-4n6h krb5kdc[20183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:12:43.456545 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:43.435360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:42371 (local address 127.6.53.65:59935)
0620 07:12:43.435685 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:43.435721 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:43.436325 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:43.440301 (+ 3976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:43.440312 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:43.441055 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:43.441784 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:43.441803 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:43.443376 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:43.443379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:43.444768 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:43.444780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:43.445711 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:43.445718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:43.445961 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:43.446880 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:43.446896 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:43.449671 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:43.453197 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:43.453205 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:43.453209 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:43.453647 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:43.454072 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:43.454077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:43.454081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:43.454313 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:43.455010 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:43.455015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:43.455225 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 07:12:43.455495 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 07:12:43.456051 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":106,"spinlock_wait_cycles":4608}
I20260620 07:12:43.456507 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:43.435796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:59935 (local address 127.6.53.67:42371)
0620 07:12:43.437681 (+ 1885us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:43.437696 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:43.437740 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:12:43.439525 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:43.439533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:43.439872 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:43.440175 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:43.441976 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:43.443191 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:43.444923 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:43.445506 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:43.445723 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:43.449930 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:43.449937 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:43.449948 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:43.449983 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:43.453058 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:43.453794 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:43.453800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:43.453805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:43.453953 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:43.454423 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:43.454429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:43.454433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:43.454880 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:43.455054 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:43.455695 (+ 641us) server_negotiation.cc:299] Negotiation successful
0620 07:12:43.456049 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1684,"thread_start_us":1494,"threads_started":1}
I20260620 07:12:43.456595 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:43.435727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:33309 (local address 127.6.53.66:42329)
0620 07:12:43.437762 (+ 2035us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:43.437772 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:43.437801 (+ 29us) server_negotiation.cc:407] Connection header received
0620 07:12:43.439525 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:43.439534 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:43.439874 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:43.440172 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:43.441974 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:43.443171 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:43.444895 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:43.445511 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:43.445730 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:43.449618 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:43.449630 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:43.449657 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:43.449716 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:43.452862 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:43.453629 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:43.453638 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:43.453645 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:43.453811 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:43.454278 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:43.454285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:43.454288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:43.454787 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:43.454973 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:43.455697 (+ 724us) server_negotiation.cc:299] Negotiation successful
0620 07:12:43.456033 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1903,"thread_start_us":1792,"threads_started":1}
I20260620 07:12:43.456652 21764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:43.435448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:42329 (local address 127.6.53.65:33309)
0620 07:12:43.437225 (+ 1777us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:43.437246 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:43.437356 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:43.440302 (+ 2946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:43.440312 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:43.441055 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:43.441783 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:43.441803 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:43.443364 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:43.443373 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:43.444762 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:43.444774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:43.445752 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:43.445760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:43.445965 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:43.446697 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:43.446724 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:43.449316 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:43.453025 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:43.453035 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:43.453057 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:43.453491 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:43.453933 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:43.453938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:43.453942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:43.454143 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:43.454926 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:43.454934 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:43.455225 (+ 291us) client_negotiation.cc:769] Sending connection context
0620 07:12:43.455494 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 07:12:43.456223 (+ 729us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1616,"mutex_wait_us":407,"spinlock_wait_cycles":28672,"thread_start_us":1440,"threads_started":1}
W20260620 07:12:43.460155 21384 leader_election.cc:341] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371): Not found: Tablet not found: bf04470b13b7463cbdaa4884808da1f8
W20260620 07:12:43.460412 21382 leader_election.cc:341] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 393a36edb57f4b888fd840074d1a189c (127.6.53.66:42329): Not found: Tablet not found: bf04470b13b7463cbdaa4884808da1f8
I20260620 07:12:43.460582 21382 leader_election.cc:302] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b81a89863b5448f1a3d0c22a04cd49a7; no voters: 393a36edb57f4b888fd840074d1a189c, c97850cb3bba4ffd817f576a179018c4
I20260620 07:12:43.460979 21763 raft_consensus.cc:2726] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:12:43.691816 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:43.690818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:35139)
0620 07:12:43.691299 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:43.691485 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":256,"threads_started":1}
I20260620 07:12:44.115439 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:44.114533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:54941)
0620 07:12:44.115122 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:44.115231 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":194,"threads_started":1}
I20260620 07:12:44.254112 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:44.253432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:33887)
0620 07:12:44.253879 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:44.253956 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":238,"threads_started":1}
I20260620 07:12:44.693315 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:44.692614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:55393)
0620 07:12:44.693089 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:44.693168 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":281,"threads_started":1}
I20260620 07:12:45.117147 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:45.116356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:42669)
0620 07:12:45.116892 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:45.116966 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":329,"threads_started":1}
I20260620 07:12:45.255571 21772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:45.254883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:60021)
0620 07:12:45.255329 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:45.255403 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":234,"threads_started":1}
I20260620 07:12:45.502158 21773 raft_consensus.cc:482] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:45.502408 21773 raft_consensus.cc:504] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:45.503021 21773 leader_election.cc:288] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371), 393a36edb57f4b888fd840074d1a189c (127.6.53.66:42329)
W20260620 07:12:45.503949 21384 leader_election.cc:341] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371): Not found: Tablet not found: bf04470b13b7463cbdaa4884808da1f8
W20260620 07:12:45.504146 21382 leader_election.cc:341] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 393a36edb57f4b888fd840074d1a189c (127.6.53.66:42329): Not found: Tablet not found: bf04470b13b7463cbdaa4884808da1f8
I20260620 07:12:45.504257 21382 leader_election.cc:302] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b81a89863b5448f1a3d0c22a04cd49a7; no voters: 393a36edb57f4b888fd840074d1a189c, c97850cb3bba4ffd817f576a179018c4
I20260620 07:12:45.504444 21773 raft_consensus.cc:2726] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:12:45.694932 21774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:45.694169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:59635)
0620 07:12:45.694688 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:45.694763 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":273,"threads_started":1}
I20260620 07:12:46.118772 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:46.118022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:42891)
0620 07:12:46.118506 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:46.118603 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":284,"threads_started":1}
I20260620 07:12:46.257042 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:46.256399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:34207)
0620 07:12:46.256802 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:46.256880 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":209,"threads_started":1}
I20260620 07:12:46.696406 21777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:46.695745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:51913)
0620 07:12:46.696150 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:46.696233 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":194,"threads_started":1}
I20260620 07:12:47.120446 21778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:47.119709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:39409)
0620 07:12:47.120160 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:47.120250 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":208,"threads_started":1}
I20260620 07:12:47.258961 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:47.258051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:41103)
0620 07:12:47.258682 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:47.258777 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":339,"threads_started":1}
I20260620 07:12:47.697993 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:47.697290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:42237)
0620 07:12:47.697758 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:47.697841 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":244,"threads_started":1}
I20260620 07:12:47.841979 21781 raft_consensus.cc:482] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:47.842211 21781 raft_consensus.cc:504] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:47.843254 21781 leader_election.cc:288] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371), 393a36edb57f4b888fd840074d1a189c (127.6.53.66:42329)
W20260620 07:12:47.843860 21384 leader_election.cc:341] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371): Not found: Tablet not found: bf04470b13b7463cbdaa4884808da1f8
W20260620 07:12:47.844244 21382 leader_election.cc:341] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 393a36edb57f4b888fd840074d1a189c (127.6.53.66:42329): Not found: Tablet not found: bf04470b13b7463cbdaa4884808da1f8
I20260620 07:12:47.844368 21382 leader_election.cc:302] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b81a89863b5448f1a3d0c22a04cd49a7; no voters: 393a36edb57f4b888fd840074d1a189c, c97850cb3bba4ffd817f576a179018c4
I20260620 07:12:47.844617 21781 raft_consensus.cc:2726] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:12:48.122022 21782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:48.121352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:36867)
0620 07:12:48.121791 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:48.121873 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":235,"threads_started":1}
I20260620 07:12:48.260798 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:48.259927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:59639)
0620 07:12:48.260511 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:48.260615 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":346,"threads_started":1}
I20260620 07:12:48.699651 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:48.698862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:40739)
0620 07:12:48.699385 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:48.699478 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":292,"threads_started":1}
I20260620 07:12:49.123692 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:49.122943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:39747)
0620 07:12:49.123409 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:49.123486 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":264,"threads_started":1}
I20260620 07:12:49.262348 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:49.261592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:41819)
0620 07:12:49.262001 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:49.262086 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":221,"threads_started":1}
I20260620 07:12:49.701306 21787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:49.700506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:36415)
0620 07:12:49.701030 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:49.701133 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":262,"threads_started":1}
I20260620 07:12:50.125329 21788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:50.124568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:54849)
0620 07:12:50.125058 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:50.125136 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":221,"threads_started":1}
I20260620 07:12:50.169697 21789 raft_consensus.cc:482] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:50.169867 21789 raft_consensus.cc:504] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:50.170487 21789 leader_election.cc:288] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371), 393a36edb57f4b888fd840074d1a189c (127.6.53.66:42329)
W20260620 07:12:50.171365 21384 leader_election.cc:341] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371): Not found: Tablet not found: bf04470b13b7463cbdaa4884808da1f8
W20260620 07:12:50.171593 21382 leader_election.cc:341] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 393a36edb57f4b888fd840074d1a189c (127.6.53.66:42329): Not found: Tablet not found: bf04470b13b7463cbdaa4884808da1f8
I20260620 07:12:50.171710 21382 leader_election.cc:302] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b81a89863b5448f1a3d0c22a04cd49a7; no voters: 393a36edb57f4b888fd840074d1a189c, c97850cb3bba4ffd817f576a179018c4
I20260620 07:12:50.172001 21789 raft_consensus.cc:2726] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:12:50.264060 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:50.263159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:49513)
0620 07:12:50.263711 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:50.263806 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":275,"threads_started":1}
I20260620 07:12:50.702893 21791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:50.702197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:44609)
0620 07:12:50.702622 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:50.702717 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":249,"threads_started":1}
I20260620 07:12:51.126871 21792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:51.126118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:47997)
0620 07:12:51.126630 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:51.126702 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":327,"threads_started":1}
I20260620 07:12:51.265599 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:51.264919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:41047)
0620 07:12:51.265342 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:51.265418 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":241,"threads_started":1}
I20260620 07:12:51.704479 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:51.703807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:55025)
0620 07:12:51.704251 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:51.704329 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":219,"threads_started":1}
I20260620 07:12:52.128454 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:52.127700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:35321)
0620 07:12:52.128219 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:52.128297 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":275,"threads_started":1}
I20260620 07:12:52.267297 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:52.266627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:53843)
0620 07:12:52.267037 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:52.267114 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":229,"threads_started":1}
I20260620 07:12:52.706012 21797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:52.705339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:44907)
0620 07:12:52.705752 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:52.705825 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":239,"threads_started":1}
I20260620 07:12:53.129904 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:53.129229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:57139)
0620 07:12:53.129642 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:53.129716 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":219,"threads_started":1}
I20260620 07:12:53.269083 21799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:53.268365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:48073)
0620 07:12:53.268810 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:53.268902 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":251,"threads_started":1}
I20260620 07:12:53.707705 21800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:53.706920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:34599)
0620 07:12:53.707391 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:53.707525 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":230,"threads_started":1}
I20260620 07:12:53.761256 21801 raft_consensus.cc:482] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:53.761477 21801 raft_consensus.cc:504] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97850cb3bba4ffd817f576a179018c4" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 42371 } } peers { permanent_uuid: "b81a89863b5448f1a3d0c22a04cd49a7" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 35981 } } peers { permanent_uuid: "393a36edb57f4b888fd840074d1a189c" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42329 } }
I20260620 07:12:53.762176 21801 leader_election.cc:288] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371), 393a36edb57f4b888fd840074d1a189c (127.6.53.66:42329)
W20260620 07:12:53.763190 21384 leader_election.cc:341] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c97850cb3bba4ffd817f576a179018c4 (127.6.53.67:42371): Not found: Tablet not found: bf04470b13b7463cbdaa4884808da1f8
W20260620 07:12:53.763412 21382 leader_election.cc:341] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 393a36edb57f4b888fd840074d1a189c (127.6.53.66:42329): Not found: Tablet not found: bf04470b13b7463cbdaa4884808da1f8
I20260620 07:12:53.763528 21382 leader_election.cc:302] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b81a89863b5448f1a3d0c22a04cd49a7; no voters: 393a36edb57f4b888fd840074d1a189c, c97850cb3bba4ffd817f576a179018c4
I20260620 07:12:53.763772 21801 raft_consensus.cc:2726] T bf04470b13b7463cbdaa4884808da1f8 P b81a89863b5448f1a3d0c22a04cd49a7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:12:54.131373 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:54.130692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:39101)
0620 07:12:54.131130 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:54.131226 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":220,"threads_started":1}
I20260620 07:12:54.272929 21803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:54.269893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:52411)
0620 07:12:54.272577 (+ 2684us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:54.272691 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2449,"thread_start_us":2343,"threads_started":1}
I20260620 07:12:54.709440 21804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:54.708639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:43395)
0620 07:12:54.709133 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:54.709237 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":252,"threads_started":1}
I20260620 07:12:55.133050 21805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:55.132251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:48819)
0620 07:12:55.132725 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:55.132835 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":255,"threads_started":1}
I20260620 07:12:55.274588 21806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:55.273780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:51571)
0620 07:12:55.274216 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:55.274364 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":230,"threads_started":1}
I20260620 07:12:55.711198 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:55.710413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:59611)
0620 07:12:55.710930 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:55.711035 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":273,"threads_started":1}
I20260620 07:12:56.134728 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:56.133904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:42017)
0620 07:12:56.134436 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:56.134540 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":258,"threads_started":1}
I20260620 07:12:56.276175 21809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:56.275482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:34753)
0620 07:12:56.275911 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:56.275998 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":227,"threads_started":1}
I20260620 07:12:56.712828 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:56.712099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:55537)
0620 07:12:56.712558 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:56.712637 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":214,"threads_started":1}
I20260620 07:12:57.136590 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:57.135762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:33773)
0620 07:12:57.136303 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:57.136397 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":294,"threads_started":1}
I20260620 07:12:57.278018 21812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:57.277309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.65:54669)
0620 07:12:57.277746 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:57.277840 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":211,"threads_started":1}
I20260620 07:12:57.714808 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:57.714002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.66:52477)
0620 07:12:57.714523 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:57.714644 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":293,"threads_started":1}
I20260620 07:12:58.120807 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 21332
I20260620 07:12:58.136315 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:58.121907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.6.53.65:35981)
0620 07:12:58.122509 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:58.122539 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:58.122785 (+ 246us) server_negotiation.cc:407] Connection header received
0620 07:12:58.123043 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:58.123052 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:58.123312 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:58.123500 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:58.124976 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:58.125978 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:58.127274 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:58.127915 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:58.128053 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:58.130794 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:58.130805 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:58.130818 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:58.130851 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:58.133496 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:58.134176 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:58.134182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:58.134184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:58.134386 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:58.134754 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:58.134759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:58.134762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:58.135111 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:58.135451 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:58.135721 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 07:12:58.136007 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":249,"threads_started":1}
I20260620 07:12:58.137564 21490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:58.138695 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:58.137598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45559 (local address 127.6.53.67:55227)
0620 07:12:58.138208 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:58.138416 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:45559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":264,"threads_started":1}
I20260620 07:12:58.229467 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 21332
I20260620 07:12:58.246346 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 21499
I20260620 07:12:58.259670 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:58.247058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.6.53.66:42329)
0620 07:12:58.247520 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:58.247524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:58.247538 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:12:58.247590 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:58.247594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:58.247680 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:58.247799 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:58.249139 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:58.249923 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:58.251235 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:58.251643 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:58.251767 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:58.254340 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:58.254345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:58.254348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:58.254380 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:58.256927 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:58.257613 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:58.257620 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:58.257624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:58.257764 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:58.258116 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:58.258123 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:58.258126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:58.258515 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:58.258782 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:58.259066 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 07:12:58.259284 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":220,"threads_started":1}
I20260620 07:12:58.260334 21622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:58.348961 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 21499
I20260620 07:12:58.365649 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 21630
I20260620 07:12:58.378811 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:58.366567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.6.53.67:42371)
0620 07:12:58.367175 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:58.367180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:58.367197 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:12:58.367253 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:58.367256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:58.367327 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:58.367427 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:58.368829 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:58.369671 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:58.370879 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:58.371285 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:58.371383 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:58.373829 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:58.373834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:58.373837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:58.373866 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:58.376466 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:58.377064 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:58.377068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:58.377070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:58.377196 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:58.377491 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:58.377494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:58.377495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:58.377733 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:58.377952 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:58.378154 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 07:12:58.378514 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":339,"threads_started":1}
I20260620 07:12:58.379606 21753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:12:58.467296 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 21630
2026-06-20T07:12:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23958 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:12:58 dist-test-slave-4n6h krb5kdc[21823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:12:58 dist-test-slave-4n6h krb5kdc[21823](info): set up 2 sockets
Jun 20 07:12:58 dist-test-slave-4n6h krb5kdc[21823](info): commencing operation
krb5kdc: starting...
W20260620 07:12:59.563014 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:12:59 dist-test-slave-4n6h krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:12:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:12:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:12:59.795359 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:41535
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:41535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:13:00.137557 21836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:13:00.137858 21836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:13:00.138000 21836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:13:00.145401 21836 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 07:13:00.145495 21836 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 07:13:00.145583 21836 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 07:13:00.145691 21836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:13:00.145788 21836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:13:00.145915 21836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:13:00.146013 21836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:13:00.146101 21836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:13:00.156384 21836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:41535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:41535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:13:00.157716 21836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:13:00.159067 21836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:13:00.171506 21841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:00.171808 21836 server_base.cc:1034] running on GCE node
W20260620 07:13:00.171533 21842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:00.171550 21844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:00.172960 21836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:13:00.174352 21836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:13:00.175568 21836 hybrid_clock.cc:648] HybridClock initialized: now 1781939580175514 us; error 29 us; skew 500 ppm
I20260620 07:13:00.175858 21836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:13:00 dist-test-slave-4n6h krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939580, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:13:00.183147 21836 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:13:00.184825 21836 webserver.cc:460] Webserver started at http://127.6.53.126:45807/ using document root <none> and password file <none>
I20260620 07:13:00.185564 21836 fs_manager.cc:359] Metadata directory not provided
I20260620 07:13:00.185684 21836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:13:00.186048 21836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:13:00.189159 21836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "b5505e04949e465d97a722d359ce9ec8"
format_stamp: "Formatted at 2026-06-20 07:13:00 on dist-test-slave-4n6h"
server_key: "afc4a5dcf67c6da4a35da16adac3aa55"
server_key_iv: "49994c063214bf95477b09c69a4d56b6"
server_key_version: "encryptionkey@0"
I20260620 07:13:00.189975 21836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "b5505e04949e465d97a722d359ce9ec8"
format_stamp: "Formatted at 2026-06-20 07:13:00 on dist-test-slave-4n6h"
server_key: "afc4a5dcf67c6da4a35da16adac3aa55"
server_key_iv: "49994c063214bf95477b09c69a4d56b6"
server_key_version: "encryptionkey@0"
I20260620 07:13:00.195266 21836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 07:13:00.198875 21851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:00.200222 21836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 07:13:00.200415 21836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "b5505e04949e465d97a722d359ce9ec8"
format_stamp: "Formatted at 2026-06-20 07:13:00 on dist-test-slave-4n6h"
server_key: "afc4a5dcf67c6da4a35da16adac3aa55"
server_key_iv: "49994c063214bf95477b09c69a4d56b6"
server_key_version: "encryptionkey@0"
I20260620 07:13:00.200655 21836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:13:00.239666 21836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:13:00.243736 21836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:13:00.244012 21836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:13:00.262900 21836 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:41535
I20260620 07:13:00.262908 21902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:41535 every 8 connection(s)
I20260620 07:13:00.264271 21836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:13:00.267889 21903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:13:00.268452 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 21836
I20260620 07:13:00.268674 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:13:00.269522 6357 external_mini_cluster.cc:1428] Setting key 85ee8ff6dc56478e89778b40f0e9807f
I20260620 07:13:00.282063 21903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Bootstrap starting.
Jun 20 07:13:00 dist-test-slave-4n6h krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:13:00.285750 21903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Neither blocks nor log segments found. Creating new log.
I20260620 07:13:00.287096 21903 log.cc:824] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:13:00.290357 21903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: No bootstrap required, opened a new log
I20260620 07:13:00.294046 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:00.271624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.6.53.126:41535)
0620 07:13:00.272421 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:00.272432 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:00.272471 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:13:00.274236 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:00.274326 (+ 90us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:00.274696 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:00.275009 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:00.276844 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:00.278114 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:00.280080 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:00.280727 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:00.280850 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:00.285667 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:00.285678 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:00.285696 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:00.285737 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:00.289649 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:00.290468 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:00.290484 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:00.290492 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:00.290667 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:00.291130 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:00.291142 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:00.291146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:00.291674 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:00.292070 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:00.292568 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 07:13:00.293022 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":371,"threads_started":1}
I20260620 07:13:00.298373 21903 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } }
I20260620 07:13:00.298740 21903 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:13:00.298846 21903 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5505e04949e465d97a722d359ce9ec8, State: Initialized, Role: FOLLOWER
I20260620 07:13:00.299443 21903 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } }
I20260620 07:13:00.299759 21903 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:13:00.299918 21903 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:13:00.300112 21903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:13:00.302217 21903 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } }
I20260620 07:13:00.302775 21903 leader_election.cc:302] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5505e04949e465d97a722d359ce9ec8; no voters:
I20260620 07:13:00.303483 21903 leader_election.cc:288] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:13:00.303623 21908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:13:00.304404 21908 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 1 LEADER]: Becoming Leader. State: Replica: b5505e04949e465d97a722d359ce9ec8, State: Running, Role: LEADER
I20260620 07:13:00.304983 21908 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } }
I20260620 07:13:00.305841 21903 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:13:00.307876 21909 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5505e04949e465d97a722d359ce9ec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } } }
I20260620 07:13:00.308236 21909 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:13:00.308928 21910 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5505e04949e465d97a722d359ce9ec8. Latest consensus state: current_term: 1 leader_uuid: "b5505e04949e465d97a722d359ce9ec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } } }
I20260620 07:13:00.309244 21910 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:13:00.311153 21914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:13:00.316283 21914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:13:00.325698 21914 catalog_manager.cc:1347] Generated new cluster ID: 4f5c2c9414ba41938cabda84525d68f2
I20260620 07:13:00.325832 21914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:13:00.337070 21914 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:13:00.338380 21914 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:13:00.344734 21914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Generated new TSK 0
I20260620 07:13:00.345506 21914 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:13:00.418013 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41535
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:13:00.752429 21929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:13:00.752707 21929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:13:00.752806 21929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:13:00.760016 21929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:13:00.760131 21929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:13:00.760243 21929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:13:00.770147 21929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:13:00.771441 21929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:13:00.772681 21929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:13:00.786064 21934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:00.786127 21937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:00.786162 21935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:00.786863 21929 server_base.cc:1034] running on GCE node
I20260620 07:13:00.787497 21929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:13:00.788444 21929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:13:00.789673 21929 hybrid_clock.cc:648] HybridClock initialized: now 1781939580789623 us; error 44 us; skew 500 ppm
I20260620 07:13:00.790014 21929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:13:00 dist-test-slave-4n6h krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939580, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:13:00.793638 21929 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:13:00.795159 21929 webserver.cc:460] Webserver started at http://127.6.53.65:41457/ using document root <none> and password file <none>
I20260620 07:13:00.795845 21929 fs_manager.cc:359] Metadata directory not provided
I20260620 07:13:00.795950 21929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:13:00.796226 21929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:13:00.799269 21929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "cedbe51127774d0fb853d8da4b172ac8"
format_stamp: "Formatted at 2026-06-20 07:13:00 on dist-test-slave-4n6h"
server_key: "8d36984950388f405d3b7badb6e59ad6"
server_key_iv: "0ee35a25fdcda511ccabc85c6b6c6059"
server_key_version: "encryptionkey@0"
I20260620 07:13:00.799986 21929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "cedbe51127774d0fb853d8da4b172ac8"
format_stamp: "Formatted at 2026-06-20 07:13:00 on dist-test-slave-4n6h"
server_key: "8d36984950388f405d3b7badb6e59ad6"
server_key_iv: "0ee35a25fdcda511ccabc85c6b6c6059"
server_key_version: "encryptionkey@0"
I20260620 07:13:00.805416 21929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 07:13:00.809193 21944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:00.810627 21929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:13:00.810799 21929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "cedbe51127774d0fb853d8da4b172ac8"
format_stamp: "Formatted at 2026-06-20 07:13:00 on dist-test-slave-4n6h"
server_key: "8d36984950388f405d3b7badb6e59ad6"
server_key_iv: "0ee35a25fdcda511ccabc85c6b6c6059"
server_key_version: "encryptionkey@0"
I20260620 07:13:00.811036 21929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:13:00.849097 21929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:13:00.853718 21929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:13:00.854033 21929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:13:00.855113 21929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:13:00.856489 21929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:13:00.856603 21929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:00.856751 21929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:13:00.856828 21929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:00.890515 21929 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:42971
I20260620 07:13:00.890605 22056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:42971 every 8 connection(s)
I20260620 07:13:00.891780 21929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:13:00.892225 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 21929
I20260620 07:13:00.892481 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:13:00.893327 6357 external_mini_cluster.cc:1428] Setting key a71cb2637a12a56a771151879ccfb0fc
Jun 20 07:13:00 dist-test-slave-4n6h krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939580, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:13:00.913422 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:00.894573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:52515 (local address 127.6.53.126:41535)
0620 07:13:00.895090 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:00.895094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:00.896179 (+ 1085us) server_negotiation.cc:407] Connection header received
0620 07:13:00.897986 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:00.897994 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:00.898095 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:00.898211 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:00.899755 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:00.900927 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:00.902555 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:00.902947 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:00.903048 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:00.907584 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:00.907597 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:00.907601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:00.907635 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:00.910307 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:00.911037 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:00.911042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:00.911046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:00.911177 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:00.911626 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:00.911631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:00.911634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:00.911914 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:00.912076 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:00.912786 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 07:13:00.913037 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":295,"threads_started":1}
I20260620 07:13:00.914436 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:00.895206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:52515)
0620 07:13:00.896010 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:00.896061 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:00.897731 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:00.898491 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:00.898498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:00.899017 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:13:00.899577 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:00.899594 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:00.901089 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:00.901093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:00.902381 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:00.902393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:00.903183 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:00.903191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:00.903526 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:00.904419 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:13:00.904451 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:13:00.907331 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:13:00.910459 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:00.910468 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:00.910484 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:00.910906 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:00.911299 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:00.911305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:00.911309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:00.911503 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:00.912044 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:13:00.912052 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:13:00.912411 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 07:13:00.912673 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 07:13:00.913196 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":353,"threads_started":1}
I20260620 07:13:00.916090 22057 heartbeater.cc:347] Connected to a master server at 127.6.53.126:41535
I20260620 07:13:00.916407 22057 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
I20260620 07:13:00.917191 22057 heartbeater.cc:511] Master 127.6.53.126:41535 requested a full tablet report, sending...
I20260620 07:13:00.919332 21868 ts_manager.cc:194] Registered new tserver with Master: cedbe51127774d0fb853d8da4b172ac8 (127.6.53.65:42971)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
I20260620 07:13:00.920890 21868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:52515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:13:00.946641 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41535
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:13:01.279860 22063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:13:01.280130 22063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:13:01.280287 22063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:13:01.287796 22063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:13:01.287907 22063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:13:01.288121 22063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:13:01.298045 22063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:13:01.299361 22063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:13:01.300634 22063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:13:01.313984 22068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:01.314111 22071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:01.314134 22069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:01.315202 22063 server_base.cc:1034] running on GCE node
I20260620 07:13:01.315856 22063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:13:01.316851 22063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:13:01.318100 22063 hybrid_clock.cc:648] HybridClock initialized: now 1781939581318064 us; error 49 us; skew 500 ppm
I20260620 07:13:01.318475 22063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:13:01 dist-test-slave-4n6h krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939581, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:13:01.321604 22063 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:13:01.323122 22063 webserver.cc:460] Webserver started at http://127.6.53.66:33949/ using document root <none> and password file <none>
I20260620 07:13:01.323797 22063 fs_manager.cc:359] Metadata directory not provided
I20260620 07:13:01.323946 22063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:13:01.324250 22063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:13:01.327123 22063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "a22e4f123f9046bdae9441e9f5146271"
format_stamp: "Formatted at 2026-06-20 07:13:01 on dist-test-slave-4n6h"
server_key: "051ccf6afa29750a9515ac12ee1fb1af"
server_key_iv: "0c82811afd3444bf6f1dc3bd35389644"
server_key_version: "encryptionkey@0"
I20260620 07:13:01.327886 22063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "a22e4f123f9046bdae9441e9f5146271"
format_stamp: "Formatted at 2026-06-20 07:13:01 on dist-test-slave-4n6h"
server_key: "051ccf6afa29750a9515ac12ee1fb1af"
server_key_iv: "0c82811afd3444bf6f1dc3bd35389644"
server_key_version: "encryptionkey@0"
I20260620 07:13:01.332880 22063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:13:01.336267 22078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:01.337574 22063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 07:13:01.337751 22063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "a22e4f123f9046bdae9441e9f5146271"
format_stamp: "Formatted at 2026-06-20 07:13:01 on dist-test-slave-4n6h"
server_key: "051ccf6afa29750a9515ac12ee1fb1af"
server_key_iv: "0c82811afd3444bf6f1dc3bd35389644"
server_key_version: "encryptionkey@0"
I20260620 07:13:01.337967 22063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:13:01.354076 22063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:13:01.358589 22063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:13:01.358904 22063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:13:01.360049 22063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:13:01.361469 22063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:13:01.361575 22063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:01.361742 22063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:13:01.361819 22063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:01.396059 22063 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:45665
I20260620 07:13:01.396087 22190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:45665 every 8 connection(s)
I20260620 07:13:01.397344 22063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:13:01.399797 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 22063
I20260620 07:13:01.400019 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:13:01.400514 6357 external_mini_cluster.cc:1428] Setting key 2f36e540d0035f20bf3f8638c4359b85
Jun 20 07:13:01 dist-test-slave-4n6h krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939581, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:13:01.421073 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:01.399939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:55469 (local address 127.6.53.126:41535)
0620 07:13:01.400173 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:01.400182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:01.401457 (+ 1275us) server_negotiation.cc:407] Connection header received
0620 07:13:01.403788 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:01.403797 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:01.403970 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:01.404059 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:01.406164 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:01.407062 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:01.409601 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:01.410029 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:01.410126 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:01.413769 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:01.413779 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:01.413787 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:01.413834 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:01.417607 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:01.418455 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:01.418464 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:01.418470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:01.418638 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:01.419071 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:01.419078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:01.419084 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:01.419483 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:01.419645 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:01.420341 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 07:13:01.420605 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:13:01.422183 22193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:01.400420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:55469)
0620 07:13:01.401276 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:01.401312 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:01.403469 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:01.404353 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:01.404367 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:01.405173 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:13:01.405954 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:01.405981 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:01.407228 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:01.407236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:01.409450 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:01.409468 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:01.410288 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:01.410299 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:01.410630 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:01.411392 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:13:01.411416 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:13:01.413539 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:13:01.417741 (+ 4202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:01.417749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:01.417762 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:01.418339 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:01.418757 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:01.418760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:01.418763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:01.418941 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:01.419608 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:13:01.419614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:13:01.419936 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 07:13:01.420186 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 07:13:01.420774 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":362,"threads_started":1}
I20260620 07:13:01.423681 22191 heartbeater.cc:347] Connected to a master server at 127.6.53.126:41535
I20260620 07:13:01.424054 22191 heartbeater.cc:464] Registering TS with master...
I20260620 07:13:01.425025 22191 heartbeater.cc:511] Master 127.6.53.126:41535 requested a full tablet report, sending...
I20260620 07:13:01.426836 21868 ts_manager.cc:194] Registered new tserver with Master: a22e4f123f9046bdae9441e9f5146271 (127.6.53.66:45665)
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
I20260620 07:13:01.428018 21868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:55469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:13:01.457142 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41535
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:13:01.788973 22196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:13:01.789299 22196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:13:01.789474 22196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:13:01.796824 22196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:13:01.796993 22196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:13:01.797170 22196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:13:01.807514 22196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:13:01.808974 22196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:13:01.810236 22196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:13:01.825155 22202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:01.825193 22201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:01.825217 22204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:01.826342 22196 server_base.cc:1034] running on GCE node
I20260620 07:13:01.826934 22196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:13:01.827875 22196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:13:01.829105 22196 hybrid_clock.cc:648] HybridClock initialized: now 1781939581829065 us; error 40 us; skew 500 ppm
I20260620 07:13:01.829416 22196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:13:01 dist-test-slave-4n6h krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939581, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:13:01.832500 22196 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:13:01.834018 22196 webserver.cc:460] Webserver started at http://127.6.53.67:42879/ using document root <none> and password file <none>
I20260620 07:13:01.834851 22196 fs_manager.cc:359] Metadata directory not provided
I20260620 07:13:01.835006 22196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:13:01.835314 22196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:13:01.839954 22196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "ea601c5dde2949a5b97dab19cacd2bf9"
format_stamp: "Formatted at 2026-06-20 07:13:01 on dist-test-slave-4n6h"
server_key: "4c62ea3562631c705609db1eea49edd9"
server_key_iv: "5b2c67f4dcfdcf833631bcce617a2937"
server_key_version: "encryptionkey@0"
I20260620 07:13:01.841025 22196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "ea601c5dde2949a5b97dab19cacd2bf9"
format_stamp: "Formatted at 2026-06-20 07:13:01 on dist-test-slave-4n6h"
server_key: "4c62ea3562631c705609db1eea49edd9"
server_key_iv: "5b2c67f4dcfdcf833631bcce617a2937"
server_key_version: "encryptionkey@0"
I20260620 07:13:01.846820 22196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:13:01.850237 22211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:01.851625 22196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:13:01.851812 22196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "ea601c5dde2949a5b97dab19cacd2bf9"
format_stamp: "Formatted at 2026-06-20 07:13:01 on dist-test-slave-4n6h"
server_key: "4c62ea3562631c705609db1eea49edd9"
server_key_iv: "5b2c67f4dcfdcf833631bcce617a2937"
server_key_version: "encryptionkey@0"
I20260620 07:13:01.852033 22196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:13:01.880851 22196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:13:01.885176 22196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:13:01.885494 22196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:13:01.886528 22196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:13:01.887933 22196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:13:01.888036 22196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:01.888187 22196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:13:01.888260 22196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:01.923482 22196 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:35853
I20260620 07:13:01.923537 22323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:35853 every 8 connection(s)
I20260620 07:13:01.923838 22057 heartbeater.cc:503] Master 127.6.53.126:41535 was elected leader, sending a full tablet report...
I20260620 07:13:01.925001 22196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:13:01.930842 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 22196
I20260620 07:13:01.931020 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:13:01.931495 6357 external_mini_cluster.cc:1428] Setting key 6648c01f4849365a7c23f134c063c7f3
Jun 20 07:13:01 dist-test-slave-4n6h krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939581, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:13:01.945118 22326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:01.927581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:55391 (local address 127.6.53.126:41535)
0620 07:13:01.928038 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:01.928042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:01.929013 (+ 971us) server_negotiation.cc:407] Connection header received
0620 07:13:01.930896 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:01.930904 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:01.931008 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:01.931128 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:01.932780 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:01.933694 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:01.935380 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:01.935844 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:01.935986 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:01.939196 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:01.939202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:01.939207 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:01.939248 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:01.942171 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:01.942882 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:01.942888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:01.942891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:01.943016 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:01.943431 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:01.943436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:01.943439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:01.943697 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:01.943857 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:01.944517 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 07:13:01.944728 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":237,"threads_started":1}
I20260620 07:13:01.946192 22327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:01.928186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:55391)
0620 07:13:01.928864 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:01.928899 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:01.930619 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:01.931347 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:01.931355 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:01.931923 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:13:01.932604 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:01.932631 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:01.933840 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:01.933845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:01.935247 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:01.935260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:01.936067 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:01.936073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:01.936314 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:01.936889 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:13:01.936909 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:13:01.939006 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:13:01.942327 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:01.942336 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:01.942351 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:01.942760 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:01.943121 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:01.943127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:01.943130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:01.943308 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:01.943827 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:13:01.943834 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:13:01.944138 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 07:13:01.944383 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 07:13:01.944900 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":277,"threads_started":1}
I20260620 07:13:01.947515 22324 heartbeater.cc:347] Connected to a master server at 127.6.53.126:41535
I20260620 07:13:01.947844 22324 heartbeater.cc:464] Registering TS with master...
I20260620 07:13:01.948529 22324 heartbeater.cc:511] Master 127.6.53.126:41535 requested a full tablet report, sending...
I20260620 07:13:01.949872 21868 ts_manager.cc:194] Registered new tserver with Master: ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853)
I20260620 07:13:01.950767 21868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:55391
I20260620 07:13:01.959007 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:13:01.976540 22326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:01.963571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.6.53.126:41535)
0620 07:13:01.963740 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:01.963744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:01.964262 (+ 518us) server_negotiation.cc:407] Connection header received
0620 07:13:01.964373 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:01.964377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:01.964448 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:01.964529 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:01.965976 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:01.966959 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:01.968321 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:01.968717 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:01.968855 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:01.971417 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:01.971430 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:01.971433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:01.971464 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:01.973796 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:01.974533 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:01.974540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:01.974544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:01.974664 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:01.975041 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:01.975048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:01.975051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:01.975370 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:01.975624 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:01.975937 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 07:13:01.976177 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:13:01.980724 21868 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:13:01.987648 21868 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:13:02.019896 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:02.000133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41052 (local address 127.6.53.65:42971)
0620 07:13:02.011498 (+ 11365us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:02.011508 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:02.011534 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:13:02.011620 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:02.011627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:02.011950 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:02.012607 (+ 657us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:13:02.013214 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.014784 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.017497 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.018681 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.018841 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:02.019089 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:02.019335 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 07:13:02.019506 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11205,"thread_start_us":11091,"threads_started":1}
I20260620 07:13:02.019972 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:02.000121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42971 (local address 127.0.0.1:41052)
0620 07:13:02.001251 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:02.001298 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:02.001464 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:02.012201 (+ 10737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:02.012207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:02.012233 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:13:02.012994 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:02.013011 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.014970 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.014976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:02.017318 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:02.017332 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.018868 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.018875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:02.019072 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:02.019129 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 07:13:02.019293 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 07:13:02.019516 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":110,"thread_start_us":10500,"threads_started":1}
I20260620 07:13:02.020387 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:02.001355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41168 (local address 127.6.53.66:45665)
0620 07:13:02.010732 (+ 9377us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:02.011551 (+ 819us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:02.011597 (+ 46us) server_negotiation.cc:407] Connection header received
0620 07:13:02.011707 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:02.011714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:02.011973 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:02.012190 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:13:02.013179 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.014204 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.017240 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.018815 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.018940 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:02.019123 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:02.019868 (+ 745us) server_negotiation.cc:299] Negotiation successful
0620 07:13:02.020038 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9226,"thread_start_us":9117,"threads_started":1}
I20260620 07:13:02.020776 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:02.000608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:35853 (local address 127.0.0.1:50412)
0620 07:13:02.011135 (+ 10527us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:02.011153 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:02.011354 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:02.012139 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:02.012146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:02.012178 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:13:02.012767 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:02.012786 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.014821 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.014826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:02.017220 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:02.017232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.019381 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.019386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:02.019556 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:02.019687 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 07:13:02.019777 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 07:13:02.019925 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10384,"mutex_wait_us":122,"thread_start_us":10214,"threads_started":1}
I20260620 07:13:02.021185 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:02.000568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:45665 (local address 127.0.0.1:41168)
0620 07:13:02.010941 (+ 10373us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:02.010969 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:02.011120 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:02.012431 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:02.012438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:02.012471 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:13:02.013034 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:02.013043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.014374 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.014382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:02.017045 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:02.017062 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.019369 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.019375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:02.019550 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:02.019686 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 07:13:02.020645 (+ 959us) client_negotiation.cc:240] Negotiation successful
0620 07:13:02.020812 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10208,"mutex_wait_us":204,"thread_start_us":1481,"threads_started":1}
I20260620 07:13:02.021787 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:02.001804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50412 (local address 127.6.53.67:35853)
0620 07:13:02.002468 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:02.002477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:02.011250 (+ 8773us) server_negotiation.cc:407] Connection header received
0620 07:13:02.011625 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:02.011631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:02.011898 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:02.012120 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:13:02.012994 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.014687 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.017412 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.018816 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.018940 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:02.019158 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:02.020926 (+ 1768us) server_negotiation.cc:299] Negotiation successful
0620 07:13:02.021110 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":847,"threads_started":1}
I20260620 07:13:02.024399 21992 tablet_service.cc:1467] Processing CreateTablet for tablet d0c3c97e325a47409089e8d4ee529f10 (DEFAULT_TABLE table=test-table [id=cf631f110b9b4e1082ea1174103452af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:13:02.024719 22126 tablet_service.cc:1467] Processing CreateTablet for tablet d0c3c97e325a47409089e8d4ee529f10 (DEFAULT_TABLE table=test-table [id=cf631f110b9b4e1082ea1174103452af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:13:02.024744 22259 tablet_service.cc:1467] Processing CreateTablet for tablet d0c3c97e325a47409089e8d4ee529f10 (DEFAULT_TABLE table=test-table [id=cf631f110b9b4e1082ea1174103452af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:13:02.025904 22259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c3c97e325a47409089e8d4ee529f10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:13:02.025949 21992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c3c97e325a47409089e8d4ee529f10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:13:02.025938 22126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0c3c97e325a47409089e8d4ee529f10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:13:02.036836 22343 tablet_bootstrap.cc:492] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9: Bootstrap starting.
I20260620 07:13:02.039913 22343 tablet_bootstrap.cc:654] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9: Neither blocks nor log segments found. Creating new log.
I20260620 07:13:02.040191 22344 tablet_bootstrap.cc:492] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271: Bootstrap starting.
I20260620 07:13:02.041293 22343 log.cc:824] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:13:02.044698 22344 tablet_bootstrap.cc:654] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271: Neither blocks nor log segments found. Creating new log.
I20260620 07:13:02.046265 22344 log.cc:824] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271: Log is configured to *not* fsync() on all Append() calls
I20260620 07:13:02.046947 22345 tablet_bootstrap.cc:492] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8: Bootstrap starting.
I20260620 07:13:02.049103 22343 tablet_bootstrap.cc:492] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9: No bootstrap required, opened a new log
I20260620 07:13:02.049220 22344 tablet_bootstrap.cc:492] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271: No bootstrap required, opened a new log
I20260620 07:13:02.049396 22343 ts_tablet_manager.cc:1397] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20260620 07:13:02.049440 22344 ts_tablet_manager.cc:1397] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 07:13:02.050663 22345 tablet_bootstrap.cc:654] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8: Neither blocks nor log segments found. Creating new log.
I20260620 07:13:02.051630 22345 log.cc:824] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:13:02.054481 22345 tablet_bootstrap.cc:492] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8: No bootstrap required, opened a new log
I20260620 07:13:02.054736 22345 ts_tablet_manager.cc:1397] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 07:13:02.060298 22343 raft_consensus.cc:348] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:02.060298 22344 raft_consensus.cc:348] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:02.060787 22343 raft_consensus.cc:374] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:13:02.060787 22344 raft_consensus.cc:374] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:13:02.060936 22343 raft_consensus.cc:729] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea601c5dde2949a5b97dab19cacd2bf9, State: Initialized, Role: FOLLOWER
I20260620 07:13:02.060963 22344 raft_consensus.cc:729] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22e4f123f9046bdae9441e9f5146271, State: Initialized, Role: FOLLOWER
I20260620 07:13:02.061602 22343 consensus_queue.cc:260] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:02.061602 22344 consensus_queue.cc:260] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:02.062495 22345 raft_consensus.cc:348] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:02.062994 22345 raft_consensus.cc:374] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:13:02.063138 22345 raft_consensus.cc:729] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cedbe51127774d0fb853d8da4b172ac8, State: Initialized, Role: FOLLOWER
I20260620 07:13:02.063278 22191 heartbeater.cc:503] Master 127.6.53.126:41535 was elected leader, sending a full tablet report...
I20260620 07:13:02.063323 22324 heartbeater.cc:503] Master 127.6.53.126:41535 was elected leader, sending a full tablet report...
I20260620 07:13:02.063495 22344 ts_tablet_manager.cc:1428] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260620 07:13:02.063894 22345 consensus_queue.cc:260] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:02.064172 22343 ts_tablet_manager.cc:1428] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260620 07:13:02.067058 22345 ts_tablet_manager.cc:1428] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
W20260620 07:13:02.144402 22058 tablet.cc:2369] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:13:02.149374 22192 tablet.cc:2369] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:13:02.176784 22325 tablet.cc:2369] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:13:02.306682 22349 raft_consensus.cc:482] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:13:02.307029 22349 raft_consensus.cc:504] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:02.308817 22349 leader_election.cc:288] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cedbe51127774d0fb853d8da4b172ac8 (127.6.53.65:42971), ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853)
I20260620 07:13:02.317981 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:02.309237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:41267 (local address 127.6.53.65:42971)
0620 07:13:02.309491 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:02.309500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:02.310317 (+ 817us) server_negotiation.cc:407] Connection header received
0620 07:13:02.310582 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:02.310590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:02.310684 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:02.310787 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:13:02.311466 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.312596 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.315446 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.316741 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.316864 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:02.316969 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:02.317555 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 07:13:02.317700 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:13:02.317991 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:02.309816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:33767 (local address 127.6.53.67:35853)
0620 07:13:02.310040 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:02.310048 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:02.310608 (+ 560us) server_negotiation.cc:407] Connection header received
0620 07:13:02.310742 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:02.310749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:02.310880 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:02.310978 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:13:02.311643 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.312769 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.315427 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.316722 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.316859 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:02.316969 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:02.317559 (+ 590us) server_negotiation.cc:299] Negotiation successful
0620 07:13:02.317700 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:13:02.318185 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:02.309160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:35853 (local address 127.6.53.66:33767)
0620 07:13:02.310509 (+ 1349us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:02.310524 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:02.310630 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:02.311008 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:02.311012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:02.311030 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:13:02.311493 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:02.311502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.312966 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.312973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:02.315247 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:02.315263 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.317034 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.317039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:02.317349 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:02.317390 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 07:13:02.317489 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 07:13:02.317647 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1230,"mutex_wait_us":101,"thread_start_us":1017,"threads_started":1}
I20260620 07:13:02.318243 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:02.309143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42971 (local address 127.6.53.66:41267)
0620 07:13:02.310176 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:02.310199 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:02.310416 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:02.310810 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:02.310814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:02.310833 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:13:02.311301 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:02.311314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.312737 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.312741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:02.315249 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:02.315263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.317029 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.317037 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:02.317262 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:02.317393 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 07:13:02.317486 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 07:13:02.317647 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":112,"thread_start_us":675,"threads_started":1}
I20260620 07:13:02.319499 22012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c3c97e325a47409089e8d4ee529f10" candidate_uuid: "a22e4f123f9046bdae9441e9f5146271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cedbe51127774d0fb853d8da4b172ac8" is_pre_election: true
I20260620 07:13:02.319499 22279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c3c97e325a47409089e8d4ee529f10" candidate_uuid: "a22e4f123f9046bdae9441e9f5146271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" is_pre_election: true
I20260620 07:13:02.320082 22279 raft_consensus.cc:2454] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22e4f123f9046bdae9441e9f5146271 in term 0.
I20260620 07:13:02.320082 22012 raft_consensus.cc:2454] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22e4f123f9046bdae9441e9f5146271 in term 0.
I20260620 07:13:02.320853 22079 leader_election.cc:302] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a22e4f123f9046bdae9441e9f5146271, cedbe51127774d0fb853d8da4b172ac8; no voters:
I20260620 07:13:02.321307 22349 raft_consensus.cc:2781] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:13:02.321491 22349 raft_consensus.cc:482] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:13:02.321599 22349 raft_consensus.cc:3037] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:13:02.323936 22349 raft_consensus.cc:504] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:02.324592 22349 leader_election.cc:288] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [CANDIDATE]: Term 1 election: Requested vote from peers cedbe51127774d0fb853d8da4b172ac8 (127.6.53.65:42971), ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853)
I20260620 07:13:02.325062 22279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c3c97e325a47409089e8d4ee529f10" candidate_uuid: "a22e4f123f9046bdae9441e9f5146271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea601c5dde2949a5b97dab19cacd2bf9"
I20260620 07:13:02.325071 22012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0c3c97e325a47409089e8d4ee529f10" candidate_uuid: "a22e4f123f9046bdae9441e9f5146271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cedbe51127774d0fb853d8da4b172ac8"
I20260620 07:13:02.325268 22279 raft_consensus.cc:3037] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:13:02.325276 22012 raft_consensus.cc:3037] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:13:02.327445 22279 raft_consensus.cc:2454] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22e4f123f9046bdae9441e9f5146271 in term 1.
I20260620 07:13:02.327486 22012 raft_consensus.cc:2454] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22e4f123f9046bdae9441e9f5146271 in term 1.
I20260620 07:13:02.327884 22082 leader_election.cc:302] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a22e4f123f9046bdae9441e9f5146271, ea601c5dde2949a5b97dab19cacd2bf9; no voters:
I20260620 07:13:02.328222 22349 raft_consensus.cc:2781] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:13:02.328660 22349 raft_consensus.cc:686] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [term 1 LEADER]: Becoming Leader. State: Replica: a22e4f123f9046bdae9441e9f5146271, State: Running, Role: LEADER
I20260620 07:13:02.329113 22349 consensus_queue.cc:237] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:02.333519 21865 catalog_manager.cc:5495] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 reported cstate change: term changed from 0 to 1, leader changed from <none> to a22e4f123f9046bdae9441e9f5146271 (127.6.53.66). New cstate: current_term: 1 leader_uuid: "a22e4f123f9046bdae9441e9f5146271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } health_report { overall_health: HEALTHY } } }
I20260620 07:13:02.381979 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:02.376362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41176 (local address 127.6.53.66:45665)
0620 07:13:02.376580 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:02.376586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:02.376688 (+ 102us) server_negotiation.cc:407] Connection header received
0620 07:13:02.376962 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:02.376966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:02.377044 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:02.377201 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:13:02.377593 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.378361 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.379801 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.380212 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.380351 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:02.380848 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:13:02.381370 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:13:02.381545 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:02.381602 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 07:13:02.381683 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:13:02.389871 22012 raft_consensus.cc:1261] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Refusing update from remote peer a22e4f123f9046bdae9441e9f5146271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:13:02.389873 22279 raft_consensus.cc:1261] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9 [term 1 FOLLOWER]: Refusing update from remote peer a22e4f123f9046bdae9441e9f5146271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:13:02.390895 22354 consensus_queue.cc:1035] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:13:02.391223 22349 consensus_queue.cc:1035] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:13:02.405190 22357 mvcc.cc:205] Tried to move back new op lower bound from 7298824529459597312 to 7298824529223798784. Current Snapshot: MvccSnapshot[applied={T|T < 7298824529459597312}]
I20260620 07:13:02.409942 22359 mvcc.cc:205] Tried to move back new op lower bound from 7298824529459597312 to 7298824529223798784. Current Snapshot: MvccSnapshot[applied={T|T < 7298824529459597312}]
I20260620 07:13:02.426546 21867 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:13:02.426817 21867 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:13:02.430569 21867 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Sending DeleteTablet for 3 replicas of tablet d0c3c97e325a47409089e8d4ee529f10
I20260620 07:13:02.431887 21992 tablet_service.cc:1514] Processing DeleteTablet for tablet d0c3c97e325a47409089e8d4ee529f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:13:02 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:41052
I20260620 07:13:02.431927 22259 tablet_service.cc:1514] Processing DeleteTablet for tablet d0c3c97e325a47409089e8d4ee529f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:13:02 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:50412
I20260620 07:13:02.431989 22126 tablet_service.cc:1514] Processing DeleteTablet for tablet d0c3c97e325a47409089e8d4ee529f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:13:02 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:41168
I20260620 07:13:02.433239 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 21929
I20260620 07:13:02.442902 22369 tablet_replica.cc:330] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8: stopping tablet replica
I20260620 07:13:02.444082 22367 tablet_replica.cc:330] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9: stopping tablet replica
I20260620 07:13:02.444227 22369 raft_consensus.cc:2229] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:13:02.444803 22369 raft_consensus.cc:2258] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:13:02.444778 22367 raft_consensus.cc:2229] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:13:02.445405 22367 raft_consensus.cc:2258] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:13:02.445611 22368 tablet_replica.cc:330] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271: stopping tablet replica
I20260620 07:13:02.446035 22368 raft_consensus.cc:2229] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:13:02.447065 22368 raft_consensus.cc:2258] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:13:02.447665 22367 ts_tablet_manager.cc:1905] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:13:02.447665 22369 ts_tablet_manager.cc:1905] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:13:02.449122 22368 ts_tablet_manager.cc:1905] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:13:02.455106 22367 ts_tablet_manager.cc:1918] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:13:02.455379 22367 log.cc:1196] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/d0c3c97e325a47409089e8d4ee529f10
I20260620 07:13:02.456084 22367 ts_tablet_manager.cc:1939] T d0c3c97e325a47409089e8d4ee529f10 P ea601c5dde2949a5b97dab19cacd2bf9: Deleting consensus metadata
I20260620 07:13:02.456671 22368 ts_tablet_manager.cc:1918] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:13:02.456884 22368 log.cc:1196] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/d0c3c97e325a47409089e8d4ee529f10
I20260620 07:13:02.457432 22368 ts_tablet_manager.cc:1939] T d0c3c97e325a47409089e8d4ee529f10 P a22e4f123f9046bdae9441e9f5146271: Deleting consensus metadata
I20260620 07:13:02.458355 21855 catalog_manager.cc:4841] TS ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853): tablet d0c3c97e325a47409089e8d4ee529f10 (table test-table [id=cf631f110b9b4e1082ea1174103452af]) successfully deleted
I20260620 07:13:02.459197 21853 catalog_manager.cc:4841] TS a22e4f123f9046bdae9441e9f5146271 (127.6.53.66:45665): tablet d0c3c97e325a47409089e8d4ee529f10 (table test-table [id=cf631f110b9b4e1082ea1174103452af]) successfully deleted
I20260620 07:13:02.459827 22369 ts_tablet_manager.cc:1918] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:13:02.460000 22369 log.cc:1196] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/d0c3c97e325a47409089e8d4ee529f10
I20260620 07:13:02.460590 22369 ts_tablet_manager.cc:1939] T d0c3c97e325a47409089e8d4ee529f10 P cedbe51127774d0fb853d8da4b172ac8: Deleting consensus metadata
I20260620 07:13:02.462440 21852 catalog_manager.cc:4841] TS cedbe51127774d0fb853d8da4b172ac8 (127.6.53.65:42971): tablet d0c3c97e325a47409089e8d4ee529f10 (table test-table [id=cf631f110b9b4e1082ea1174103452af]) successfully deleted
Jun 20 07:13:02 dist-test-slave-4n6h krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:13:02.472163 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:02.434580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.6.53.65:42971)
0620 07:13:02.434862 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:02.434873 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:02.443716 (+ 8843us) server_negotiation.cc:407] Connection header received
0620 07:13:02.444907 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:02.444915 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:02.445016 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:02.445105 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:02.457003 (+ 11898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.458092 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.460262 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.460900 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.461059 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:02.465995 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:02.466007 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:02.466021 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:02.466061 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:02.469088 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:02.469825 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:02.469833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:02.469835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:02.469984 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:02.470489 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:02.470494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:02.470496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:02.470858 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:02.471178 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:02.471534 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 07:13:02.471753 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260620 07:13:02.472855 22052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:13:02.620362 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 21929
I20260620 07:13:02.638290 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 22063
Jun 20 07:13:02 dist-test-slave-4n6h krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:13:02.654791 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:02.639063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41182 (local address 127.6.53.66:45665)
0620 07:13:02.639263 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:02.639270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:02.639410 (+ 140us) server_negotiation.cc:407] Connection header received
0620 07:13:02.639640 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:02.639650 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:02.639719 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:02.639843 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:02.641137 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.642094 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.643705 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.644146 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.644279 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:02.648622 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:02.648633 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:02.648649 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:02.648688 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:02.651884 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:02.652595 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:02.652602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:02.652604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:02.652755 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:02.653141 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:02.653146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:02.653151 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:02.653517 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:02.653871 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:02.654164 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 07:13:02.654425 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:13:02.655620 22186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:13:02.768563 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 22063
I20260620 07:13:02.786908 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 22196
Jun 20 07:13:02 dist-test-slave-4n6h krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:13:02.802512 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:02.787610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.6.53.67:35853)
0620 07:13:02.787810 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:02.787816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:02.787836 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:13:02.788111 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:02.788116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:02.788181 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:02.788292 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:02.789575 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.790348 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.791636 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:02.792021 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:02.792159 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:02.796502 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:02.796517 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:02.796534 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:02.796574 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:02.799525 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:02.800240 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:02.800247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:02.800249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:02.800385 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:02.800769 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:02.800776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:02.800784 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:02.801173 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:02.801472 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:02.801798 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 07:13:02.802037 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:13:02.803241 22319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:13:02.893977 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 22196
I20260620 07:13:02.910640 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 21836
I20260620 07:13:02.911768 21898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:13:03.021505 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 21836
I20260620 07:13:23.042897 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:41535
--webserver_interface=127.6.53.126
--webserver_port=45807
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:41535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:13:23.359797 22375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:13:23.360085 22375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:13:23.360178 22375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:13:23.367834 22375 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 07:13:23.367928 22375 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 07:13:23.367967 22375 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 07:13:23.368017 22375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:13:23.368070 22375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:13:23.368136 22375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:13:23.368180 22375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:13:23.368232 22375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:13:23.378295 22375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:41535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:41535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=45807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:13:23.379521 22375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:13:23.380714 22375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:13:23.403448 22382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:23.403425 22381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:23.403875 22384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:23.405529 22375 server_base.cc:1034] running on GCE node
I20260620 07:13:23.406481 22375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:13:23.408028 22375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:13:23.409252 22375 hybrid_clock.cc:648] HybridClock initialized: now 1781939603409184 us; error 32 us; skew 500 ppm
I20260620 07:13:23.409602 22375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:13:23 dist-test-slave-4n6h krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939603, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:13:23.417385 22375 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:13:23.419013 22375 webserver.cc:460] Webserver started at http://127.6.53.126:45807/ using document root <none> and password file <none>
I20260620 07:13:23.419747 22375 fs_manager.cc:359] Metadata directory not provided
I20260620 07:13:23.419847 22375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:13:23.425443 22375 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:13:23.428210 22391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:23.429560 22375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 07:13:23.429708 22375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "b5505e04949e465d97a722d359ce9ec8"
format_stamp: "Formatted at 2026-06-20 07:13:00 on dist-test-slave-4n6h"
server_key: "afc4a5dcf67c6da4a35da16adac3aa55"
server_key_iv: "49994c063214bf95477b09c69a4d56b6"
server_key_version: "encryptionkey@0"
I20260620 07:13:23.430785 22375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:13:23.449046 22375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:13:23.453522 22375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:13:23.453776 22375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:13:23.471993 22375 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:41535
I20260620 07:13:23.472026 22442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:41535 every 8 connection(s)
I20260620 07:13:23.473592 22375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:13:23.475054 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 22375
I20260620 07:13:23.476046 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:42971
--local_ip_for_outbound_sockets=127.6.53.65
--tserver_master_addrs=127.6.53.126:41535
--webserver_port=41457
--webserver_interface=127.6.53.65
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:13:23.483467 22443 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:13:23.487265 22443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Bootstrap starting.
I20260620 07:13:23.506964 22443 log.cc:824] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:13:23.518877 22443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:13:23.519459 22443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Bootstrap complete.
I20260620 07:13:23.528825 22443 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } }
I20260620 07:13:23.529655 22443 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5505e04949e465d97a722d359ce9ec8, State: Initialized, Role: FOLLOWER
I20260620 07:13:23.530401 22443 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } }
I20260620 07:13:23.530652 22443 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:13:23.530793 22443 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:13:23.530983 22443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:13:23.535423 22443 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } }
I20260620 07:13:23.535953 22443 leader_election.cc:302] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5505e04949e465d97a722d359ce9ec8; no voters:
I20260620 07:13:23.536558 22443 leader_election.cc:288] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:13:23.536705 22447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:13:23.539062 22443 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:13:23.542558 22447 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 2 LEADER]: Becoming Leader. State: Replica: b5505e04949e465d97a722d359ce9ec8, State: Running, Role: LEADER
I20260620 07:13:23.543329 22447 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } }
I20260620 07:13:23.546499 22448 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b5505e04949e465d97a722d359ce9ec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } } }
I20260620 07:13:23.547169 22448 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:13:23.546510 22449 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5505e04949e465d97a722d359ce9ec8. Latest consensus state: current_term: 2 leader_uuid: "b5505e04949e465d97a722d359ce9ec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } } }
I20260620 07:13:23.547555 22449 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:13:23.555436 22454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:13:23.561676 22454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:13:23.563109 22454 catalog_manager.cc:1259] Loaded cluster ID: 4f5c2c9414ba41938cabda84525d68f2
I20260620 07:13:23.563220 22454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:13:23.566107 22454 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:13:23.567260 22454 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Loaded TSK: 0
I20260620 07:13:23.590802 22454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Generated new TSK 1
I20260620 07:13:23.591144 22454 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:13:23.838973 22445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:13:23.839468 22445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:13:23.839635 22445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:13:23.848822 22445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:13:23.848969 22445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:13:23.849180 22445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:13:23.859153 22445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:42971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=41457
--tserver_master_addrs=127.6.53.126:41535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:13:23.860575 22445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:13:23.861905 22445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:13:23.876128 22471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:23.876699 22473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:23.876724 22470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:23.878129 22445 server_base.cc:1034] running on GCE node
I20260620 07:13:23.878988 22445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:13:23.880162 22445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:13:23.881417 22445 hybrid_clock.cc:648] HybridClock initialized: now 1781939603881368 us; error 53 us; skew 500 ppm
I20260620 07:13:23.881805 22445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:13:23 dist-test-slave-4n6h krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939603, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:13:23.885826 22445 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:13:23.887570 22445 webserver.cc:460] Webserver started at http://127.6.53.65:41457/ using document root <none> and password file <none>
I20260620 07:13:23.888447 22445 fs_manager.cc:359] Metadata directory not provided
I20260620 07:13:23.888602 22445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:13:23.895331 22445 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:13:23.900651 22480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:23.902015 22445 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260620 07:13:23.902177 22445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "cedbe51127774d0fb853d8da4b172ac8"
format_stamp: "Formatted at 2026-06-20 07:13:00 on dist-test-slave-4n6h"
server_key: "8d36984950388f405d3b7badb6e59ad6"
server_key_iv: "0ee35a25fdcda511ccabc85c6b6c6059"
server_key_version: "encryptionkey@0"
I20260620 07:13:23.903162 22445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:13:23.921624 22445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:13:23.925839 22445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:13:23.926111 22445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:13:23.927120 22445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:13:23.928632 22445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:13:23.928718 22445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:23.928869 22445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:13:23.928962 22445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:23.966663 22445 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:42971
I20260620 07:13:23.966766 22592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:42971 every 8 connection(s)
I20260620 07:13:23.968068 22445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:13:23.971756 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 22445
I20260620 07:13:23.972393 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:45665
--local_ip_for_outbound_sockets=127.6.53.66
--tserver_master_addrs=127.6.53.126:41535
--webserver_port=33949
--webserver_interface=127.6.53.66
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:13:23 dist-test-slave-4n6h krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939603, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:13:23.991350 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:23.970689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:35263 (local address 127.6.53.126:41535)
0620 07:13:23.971333 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:23.971352 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:23.971860 (+ 508us) server_negotiation.cc:407] Connection header received
0620 07:13:23.974071 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:23.974090 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:23.974537 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:23.974941 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:23.976247 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:23.978298 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:23.979842 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:23.980468 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:23.980665 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:23.984086 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:23.984093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:23.984106 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:23.984141 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:23.987031 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:23.987779 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:23.987793 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:23.987798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:23.987952 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:23.988387 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:23.988393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:23.988396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:23.988852 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:23.989043 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:23.989777 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 07:13:23.990134 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":258,"threads_started":1}
I20260620 07:13:23.991351 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:23.970876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:35263)
0620 07:13:23.971667 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:23.971717 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:23.973438 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:23.975060 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:23.975067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:23.975554 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:13:23.976043 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:23.976060 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:23.978495 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:23.978501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:23.979682 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:23.979694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:23.980705 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:23.980712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:23.980990 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:23.981659 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:13:23.981680 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:13:23.983818 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:13:23.987190 (+ 3372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:23.987200 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:23.987213 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:23.987632 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:23.988076 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:23.988081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:23.988085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:23.988273 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:23.988997 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:13:23.989005 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:13:23.989317 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 07:13:23.989577 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 07:13:23.990075 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":354,"threads_started":1}
I20260620 07:13:23.994570 22593 heartbeater.cc:347] Connected to a master server at 127.6.53.126:41535
I20260620 07:13:23.994918 22593 heartbeater.cc:464] Registering TS with master...
I20260620 07:13:23.995857 22593 heartbeater.cc:511] Master 127.6.53.126:41535 requested a full tablet report, sending...
I20260620 07:13:23.998286 22408 ts_manager.cc:194] Registered new tserver with Master: cedbe51127774d0fb853d8da4b172ac8 (127.6.53.65:42971)
I20260620 07:13:24.000371 22408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:35263
W20260620 07:13:24.299048 22597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:13:24.299324 22597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:13:24.299423 22597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:13:24.306744 22597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:13:24.306856 22597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:13:24.306982 22597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:13:24.316577 22597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:45665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=33949
--tserver_master_addrs=127.6.53.126:41535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:13:24.317819 22597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:13:24.319111 22597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:13:24.332079 22605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:24.332312 22602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:24.332262 22603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:24.333252 22597 server_base.cc:1034] running on GCE node
I20260620 07:13:24.333925 22597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:13:24.334884 22597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:13:24.336071 22597 hybrid_clock.cc:648] HybridClock initialized: now 1781939604336019 us; error 50 us; skew 500 ppm
I20260620 07:13:24.336370 22597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:13:24 dist-test-slave-4n6h krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939604, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:13:24.339495 22597 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:13:24.340898 22597 webserver.cc:460] Webserver started at http://127.6.53.66:33949/ using document root <none> and password file <none>
I20260620 07:13:24.341527 22597 fs_manager.cc:359] Metadata directory not provided
I20260620 07:13:24.341686 22597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:13:24.346714 22597 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:13:24.349224 22612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:24.350544 22597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:13:24.350709 22597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "a22e4f123f9046bdae9441e9f5146271"
format_stamp: "Formatted at 2026-06-20 07:13:01 on dist-test-slave-4n6h"
server_key: "051ccf6afa29750a9515ac12ee1fb1af"
server_key_iv: "0c82811afd3444bf6f1dc3bd35389644"
server_key_version: "encryptionkey@0"
I20260620 07:13:24.351614 22597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:13:24.391376 22597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:13:24.395800 22597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:13:24.396076 22597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:13:24.397086 22597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:13:24.398440 22597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:13:24.398566 22597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:24.398685 22597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:13:24.398783 22597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:24.430694 22597 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:45665
I20260620 07:13:24.430787 22724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:45665 every 8 connection(s)
I20260620 07:13:24.432121 22597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:13:24.435549 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 22597
I20260620 07:13:24.436331 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:35853
--local_ip_for_outbound_sockets=127.6.53.67
--tserver_master_addrs=127.6.53.126:41535
--webserver_port=42879
--webserver_interface=127.6.53.67
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:13:24 dist-test-slave-4n6h krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939604, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:13:24.455307 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:24.434625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:58097 (local address 127.6.53.126:41535)
0620 07:13:24.434840 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:24.434847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:24.436250 (+ 1403us) server_negotiation.cc:407] Connection header received
0620 07:13:24.439164 (+ 2914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:24.439169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:24.439255 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:24.439345 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:24.441409 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.442091 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.444576 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.445641 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.445790 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:24.449372 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:24.449378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:24.449381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:24.449419 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:24.452129 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:24.452932 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:24.452940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:24.452948 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:24.453066 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:24.453519 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:24.453523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:24.453525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:24.453799 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:24.453959 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:24.454717 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 07:13:24.454953 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:13:24.456370 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:24.435153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:58097)
0620 07:13:24.436068 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:24.436114 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:24.438864 (+ 2750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:24.439613 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:24.439627 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:24.440412 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:13:24.441216 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:24.441243 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.442282 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.442290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:24.444408 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:24.444425 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.445882 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.445887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:24.446093 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:24.446857 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:13:24.446881 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:13:24.449113 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:13:24.452321 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:24.452331 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:24.452348 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:24.452796 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:24.453182 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:24.453187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:24.453191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:24.453390 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:24.453932 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:13:24.453941 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:13:24.454349 (+ 408us) client_negotiation.cc:769] Sending connection context
0620 07:13:24.454620 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 07:13:24.455163 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":307,"threads_started":1}
I20260620 07:13:24.458069 22725 heartbeater.cc:347] Connected to a master server at 127.6.53.126:41535
I20260620 07:13:24.458444 22725 heartbeater.cc:464] Registering TS with master...
I20260620 07:13:24.459151 22725 heartbeater.cc:511] Master 127.6.53.126:41535 requested a full tablet report, sending...
I20260620 07:13:24.460713 22408 ts_manager.cc:194] Registered new tserver with Master: a22e4f123f9046bdae9441e9f5146271 (127.6.53.66:45665)
I20260620 07:13:24.462100 22408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:58097
W20260620 07:13:24.768102 22728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:13:24.768353 22728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:13:24.768445 22728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:13:24.775521 22728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:13:24.775624 22728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:13:24.775748 22728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:13:24.785058 22728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:35853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=42879
--tserver_master_addrs=127.6.53.126:41535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:13:24.786201 22728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:13:24.787420 22728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:13:24.799883 22733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:24.799891 22734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:24.800029 22736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:24.801302 22728 server_base.cc:1034] running on GCE node
I20260620 07:13:24.801851 22728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:13:24.802810 22728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:13:24.803998 22728 hybrid_clock.cc:648] HybridClock initialized: now 1781939604803967 us; error 28 us; skew 500 ppm
I20260620 07:13:24.804296 22728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:13:24 dist-test-slave-4n6h krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939604, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:13:24.807376 22728 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:13:24.808827 22728 webserver.cc:460] Webserver started at http://127.6.53.67:42879/ using document root <none> and password file <none>
I20260620 07:13:24.809463 22728 fs_manager.cc:359] Metadata directory not provided
I20260620 07:13:24.809616 22728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:13:24.814553 22728 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:13:24.817039 22743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:24.818347 22728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:13:24.818513 22728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "ea601c5dde2949a5b97dab19cacd2bf9"
format_stamp: "Formatted at 2026-06-20 07:13:01 on dist-test-slave-4n6h"
server_key: "4c62ea3562631c705609db1eea49edd9"
server_key_iv: "5b2c67f4dcfdcf833631bcce617a2937"
server_key_version: "encryptionkey@0"
I20260620 07:13:24.819437 22728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:13:24.834189 22728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:13:24.839947 22728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:13:24.840278 22728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:13:24.841480 22728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:13:24.843001 22728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:13:24.843127 22728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:24.843254 22728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:13:24.843358 22728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:24.880450 22728 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:35853
I20260620 07:13:24.880542 22855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:35853 every 8 connection(s)
I20260620 07:13:24.881853 22728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:13:24.882488 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 22728
Jun 20 07:13:24 dist-test-slave-4n6h krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939604, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:13:24.899122 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:24.883610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.6.53.126:41535)
0620 07:13:24.883867 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:24.883875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:24.884264 (+ 389us) server_negotiation.cc:407] Connection header received
0620 07:13:24.884571 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:24.884582 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:24.884693 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:24.884783 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:24.886974 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.887726 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.889176 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.889691 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.889864 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:24.893075 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:24.893082 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:24.893087 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:24.893158 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:24.896405 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:24.897239 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:24.897245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:24.897248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:24.897398 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:24.897735 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:24.897741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:24.897744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:24.898070 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:24.898373 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:24.898689 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 07:13:24.898839 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":69}
I20260620 07:13:24.907119 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:24.884773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:41645 (local address 127.6.53.126:41535)
0620 07:13:24.885217 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:24.885228 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:24.889497 (+ 4269us) server_negotiation.cc:407] Connection header received
0620 07:13:24.889565 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:24.889573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:24.889661 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:24.889790 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:24.892337 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.893056 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.895476 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.895928 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.896027 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:24.900351 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:24.900359 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:24.900366 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:24.900405 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:24.904282 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:24.904962 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:24.904966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:24.904968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:24.905069 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:24.905422 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:24.905425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:24.905427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:24.905660 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:24.905771 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:24.906398 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 07:13:24.906709 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":364,"thread_start_us":251,"threads_started":1}
I20260620 07:13:24.907938 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:24.885258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:41645)
0620 07:13:24.886185 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:24.886240 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:24.889104 (+ 2864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:24.890586 (+ 1482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:24.890598 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:24.891343 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:13:24.892124 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:24.892164 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.893203 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.893210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:24.895334 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:24.895349 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.896150 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.896157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:24.896465 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:24.897090 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:13:24.897110 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:13:24.900078 (+ 2968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:13:24.904411 (+ 4333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:24.904417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:24.904429 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:24.904855 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:24.905156 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:24.905159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:24.905162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:24.905323 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:24.905760 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:13:24.905766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:13:24.906035 (+ 269us) client_negotiation.cc:769] Sending connection context
0620 07:13:24.906310 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 07:13:24.906812 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":367,"threads_started":1}
I20260620 07:13:24.909140 22856 heartbeater.cc:347] Connected to a master server at 127.6.53.126:41535
I20260620 07:13:24.909397 22856 heartbeater.cc:464] Registering TS with master...
I20260620 07:13:24.910006 22856 heartbeater.cc:511] Master 127.6.53.126:41535 requested a full tablet report, sending...
I20260620 07:13:24.911295 22408 ts_manager.cc:194] Registered new tserver with Master: ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853)
I20260620 07:13:24.912088 22408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:41645
I20260620 07:13:24.914062 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:13:24.920449 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:24.914968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.6.53.126:41535)
0620 07:13:24.915144 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:24.915152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:24.915515 (+ 363us) server_negotiation.cc:407] Connection header received
0620 07:13:24.915756 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:24.915760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:24.915867 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:24.916008 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:13:24.916444 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.917128 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.918670 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.919075 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.919206 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:24.919681 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:13:24.919872 (+ 191us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 07:13:24.920204 (+ 332us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49210: token expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 07:13:24.920572 22859 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49210: token expired
W20260620 07:13:24.921058 22331 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.53.126:41535)
I20260620 07:13:24.933145 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:24.921507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.6.53.126:41535)
0620 07:13:24.921657 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:24.921664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:24.921681 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:13:24.921850 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:24.921861 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:24.921910 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:24.922024 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:24.923406 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.924093 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.925425 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.925803 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.925931 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:24.928381 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:24.928386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:24.928389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:24.928416 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:24.930646 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:24.931282 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:24.931289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:24.931292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:24.931408 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:24.931772 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:24.931778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:24.931781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:24.932058 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:24.932324 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:24.932631 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 07:13:24.932844 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:13:24.936379 22408 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:13:24.943667 22408 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:13:24.968544 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:24.955398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.6.53.65:42971)
0620 07:13:24.956695 (+ 1297us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:24.956708 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:24.957477 (+ 769us) server_negotiation.cc:407] Connection header received
0620 07:13:24.957587 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:24.957598 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:24.957907 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:24.958118 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:13:24.959191 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.961321 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.965035 (+ 3714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.966709 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.966845 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:24.967041 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:24.968012 (+ 971us) server_negotiation.cc:299] Negotiation successful
0620 07:13:24.968178 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1122,"thread_start_us":1030,"threads_started":1}
I20260620 07:13:24.970196 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:24.955121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42971 (local address 127.0.0.1:48684)
0620 07:13:24.957080 (+ 1959us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:24.957116 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:24.957301 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:24.958427 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:24.958436 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:24.958471 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:13:24.958960 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:24.958977 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.961525 (+ 2548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.961534 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:24.964796 (+ 3262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:24.964819 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.967196 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.967204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:24.967433 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:24.967496 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 07:13:24.967640 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 07:13:24.967833 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1739,"thread_start_us":1611,"threads_started":1}
I20260620 07:13:24.974213 22528 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9223f8390c48618b175416318579f6 (DEFAULT_TABLE table=test-table [id=3219ce40af444a4ba97a19b76b373c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:13:24.975714 22528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9223f8390c48618b175416318579f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:13:24.979146 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:24.955433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:35853 (local address 127.0.0.1:51128)
0620 07:13:24.971062 (+ 15629us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:24.971087 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:24.971233 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:24.972553 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:24.972559 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:24.972584 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:13:24.973143 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:24.973157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.975079 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.975083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:24.976600 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:24.976613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.978434 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.978438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:24.978563 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:24.978585 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:13:24.978694 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 07:13:24.978832 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15469,"mutex_wait_us":395,"thread_start_us":15960,"threads_started":1}
I20260620 07:13:24.979506 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:24.956288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.6.53.67:35853)
0620 07:13:24.971652 (+ 15364us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:24.971672 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:24.971715 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:13:24.971834 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:24.971844 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:24.972077 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:24.972269 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:13:24.973318 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.974904 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.976759 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.977886 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.978004 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:24.978225 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:24.978753 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 07:13:24.978899 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12122,"thread_start_us":11948,"threads_started":1}
I20260620 07:13:24.980427 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:24.957424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:45665 (local address 127.0.0.1:40308)
0620 07:13:24.971670 (+ 14246us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:24.971689 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:24.971805 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:24.972577 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:24.972581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:24.972596 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:13:24.972983 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:24.973000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.975002 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.975007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:24.977171 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:24.977204 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.979634 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.979640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:24.979810 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:24.979841 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:13:24.979937 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 07:13:24.980061 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14127,"mutex_wait_us":3,"thread_start_us":16312,"threads_started":1}
I20260620 07:13:24.980641 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:24.957695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40308 (local address 127.6.53.66:45665)
0620 07:13:24.969374 (+ 11679us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:24.969387 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:24.971924 (+ 2537us) server_negotiation.cc:407] Connection header received
0620 07:13:24.972012 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:24.972018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:24.972207 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:24.972380 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:13:24.973213 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.974875 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.977400 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:24.979199 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:24.979319 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:24.979492 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:24.979991 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 07:13:24.980151 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11469,"thread_start_us":11700,"threads_started":1}
I20260620 07:13:24.983225 22791 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9223f8390c48618b175416318579f6 (DEFAULT_TABLE table=test-table [id=3219ce40af444a4ba97a19b76b373c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:13:24.983412 22660 tablet_service.cc:1467] Processing CreateTablet for tablet 6c9223f8390c48618b175416318579f6 (DEFAULT_TABLE table=test-table [id=3219ce40af444a4ba97a19b76b373c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:13:24.984436 22660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9223f8390c48618b175416318579f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:13:24.984555 22791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c9223f8390c48618b175416318579f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:13:24.992532 22872 tablet_bootstrap.cc:492] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8: Bootstrap starting.
I20260620 07:13:24.996443 22872 tablet_bootstrap.cc:654] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8: Neither blocks nor log segments found. Creating new log.
I20260620 07:13:24.996706 22873 tablet_bootstrap.cc:492] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271: Bootstrap starting.
I20260620 07:13:24.997781 22872 log.cc:824] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:13:24.999948 22875 tablet_bootstrap.cc:492] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9: Bootstrap starting.
I20260620 07:13:25.000249 22873 tablet_bootstrap.cc:654] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271: Neither blocks nor log segments found. Creating new log.
I20260620 07:13:25.000757 22872 tablet_bootstrap.cc:492] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8: No bootstrap required, opened a new log
I20260620 07:13:25.001012 22872 ts_tablet_manager.cc:1397] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260620 07:13:25.001524 22873 log.cc:824] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271: Log is configured to *not* fsync() on all Append() calls
I20260620 07:13:25.003347 22875 tablet_bootstrap.cc:654] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9: Neither blocks nor log segments found. Creating new log.
I20260620 07:13:25.003989 22593 heartbeater.cc:503] Master 127.6.53.126:41535 was elected leader, sending a full tablet report...
I20260620 07:13:25.004515 22875 log.cc:824] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:13:25.005170 22873 tablet_bootstrap.cc:492] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271: No bootstrap required, opened a new log
I20260620 07:13:25.005470 22873 ts_tablet_manager.cc:1397] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 07:13:25.010535 22872 raft_consensus.cc:348] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:25.010983 22872 raft_consensus.cc:374] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:13:25.011093 22872 raft_consensus.cc:729] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cedbe51127774d0fb853d8da4b172ac8, State: Initialized, Role: FOLLOWER
I20260620 07:13:25.011819 22872 consensus_queue.cc:260] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:25.013492 22875 tablet_bootstrap.cc:492] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9: No bootstrap required, opened a new log
I20260620 07:13:25.013743 22872 ts_tablet_manager.cc:1428] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260620 07:13:25.013760 22875 ts_tablet_manager.cc:1397] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260620 07:13:25.016973 22873 raft_consensus.cc:348] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:25.017395 22873 raft_consensus.cc:374] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:13:25.017489 22873 raft_consensus.cc:729] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22e4f123f9046bdae9441e9f5146271, State: Initialized, Role: FOLLOWER
I20260620 07:13:25.018064 22873 consensus_queue.cc:260] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:25.019102 22725 heartbeater.cc:503] Master 127.6.53.126:41535 was elected leader, sending a full tablet report...
I20260620 07:13:25.019712 22873 ts_tablet_manager.cc:1428] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 07:13:25.023623 22875 raft_consensus.cc:348] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:25.024003 22875 raft_consensus.cc:374] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:13:25.024076 22875 raft_consensus.cc:729] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea601c5dde2949a5b97dab19cacd2bf9, State: Initialized, Role: FOLLOWER
I20260620 07:13:25.024547 22875 consensus_queue.cc:260] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:25.025400 22856 heartbeater.cc:503] Master 127.6.53.126:41535 was elected leader, sending a full tablet report...
I20260620 07:13:25.025962 22875 ts_tablet_manager.cc:1428] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260620 07:13:25.080982 22879 raft_consensus.cc:482] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:13:25.081238 22879 raft_consensus.cc:504] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:25.082914 22879 leader_election.cc:288] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cedbe51127774d0fb853d8da4b172ac8 (127.6.53.65:42971), ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853)
I20260620 07:13:25.092021 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:25.083237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:55531 (local address 127.6.53.65:42971)
0620 07:13:25.083480 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:25.083489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:25.084291 (+ 802us) server_negotiation.cc:407] Connection header received
0620 07:13:25.084515 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:25.084520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:25.084622 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:25.084716 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:13:25.085423 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:25.086559 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:25.089525 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:25.090781 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:25.090911 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:25.091006 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:25.091636 (+ 630us) server_negotiation.cc:299] Negotiation successful
0620 07:13:25.091762 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:13:25.092017 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:25.083561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:35721 (local address 127.6.53.67:35853)
0620 07:13:25.083813 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:25.083822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:25.084528 (+ 706us) server_negotiation.cc:407] Connection header received
0620 07:13:25.084683 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:25.084690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:25.084777 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:25.084864 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:13:25.085531 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:25.086611 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:25.089518 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:25.090781 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:25.090911 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:25.091010 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:25.091631 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 07:13:25.091772 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:13:25.092265 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:25.083181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:35853 (local address 127.6.53.66:35721)
0620 07:13:25.084441 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:25.084454 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:25.084555 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:25.084903 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:25.084909 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:25.084941 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:13:25.085372 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:25.085385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:25.086724 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:25.086729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:25.089369 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:25.089377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:25.091069 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:25.091075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:25.091384 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:25.091488 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 07:13:25.091588 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 07:13:25.091755 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"mutex_wait_us":322,"thread_start_us":552,"threads_started":1}
I20260620 07:13:25.092448 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:25.083158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42971 (local address 127.6.53.66:55531)
0620 07:13:25.084157 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:25.084176 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:25.084363 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:25.084752 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:25.084755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:25.084774 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:13:25.085261 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:25.085278 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:25.086724 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:25.086729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:25.089360 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:25.089376 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:25.091073 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:25.091078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:25.091385 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:25.091494 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 07:13:25.091588 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 07:13:25.091755 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":218,"spinlock_wait_cycles":20352,"thread_start_us":950,"threads_started":1}
I20260620 07:13:25.093272 22811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9223f8390c48618b175416318579f6" candidate_uuid: "a22e4f123f9046bdae9441e9f5146271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" is_pre_election: true
I20260620 07:13:25.093561 22548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9223f8390c48618b175416318579f6" candidate_uuid: "a22e4f123f9046bdae9441e9f5146271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cedbe51127774d0fb853d8da4b172ac8" is_pre_election: true
I20260620 07:13:25.093796 22811 raft_consensus.cc:2454] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22e4f123f9046bdae9441e9f5146271 in term 0.
I20260620 07:13:25.094084 22548 raft_consensus.cc:2454] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22e4f123f9046bdae9441e9f5146271 in term 0.
I20260620 07:13:25.094499 22616 leader_election.cc:302] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a22e4f123f9046bdae9441e9f5146271, ea601c5dde2949a5b97dab19cacd2bf9; no voters:
I20260620 07:13:25.094909 22879 raft_consensus.cc:2781] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:13:25.095054 22879 raft_consensus.cc:482] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:13:25.095165 22879 raft_consensus.cc:3037] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:13:25.097275 22879 raft_consensus.cc:504] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:25.097826 22879 leader_election.cc:288] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [CANDIDATE]: Term 1 election: Requested vote from peers cedbe51127774d0fb853d8da4b172ac8 (127.6.53.65:42971), ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853)
I20260620 07:13:25.098215 22548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9223f8390c48618b175416318579f6" candidate_uuid: "a22e4f123f9046bdae9441e9f5146271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cedbe51127774d0fb853d8da4b172ac8"
I20260620 07:13:25.098344 22811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c9223f8390c48618b175416318579f6" candidate_uuid: "a22e4f123f9046bdae9441e9f5146271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea601c5dde2949a5b97dab19cacd2bf9"
I20260620 07:13:25.098460 22548 raft_consensus.cc:3037] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:13:25.098555 22811 raft_consensus.cc:3037] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:13:25.101717 22811 raft_consensus.cc:2454] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22e4f123f9046bdae9441e9f5146271 in term 1.
I20260620 07:13:25.101728 22548 raft_consensus.cc:2454] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22e4f123f9046bdae9441e9f5146271 in term 1.
I20260620 07:13:25.102125 22613 leader_election.cc:302] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a22e4f123f9046bdae9441e9f5146271, cedbe51127774d0fb853d8da4b172ac8; no voters:
I20260620 07:13:25.102463 22879 raft_consensus.cc:2781] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:13:25.102903 22879 raft_consensus.cc:686] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [term 1 LEADER]: Becoming Leader. State: Replica: a22e4f123f9046bdae9441e9f5146271, State: Running, Role: LEADER
I20260620 07:13:25.103375 22879 consensus_queue.cc:237] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:25.108122 22408 catalog_manager.cc:5495] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 reported cstate change: term changed from 0 to 1, leader changed from <none> to a22e4f123f9046bdae9441e9f5146271 (127.6.53.66). New cstate: current_term: 1 leader_uuid: "a22e4f123f9046bdae9441e9f5146271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } health_report { overall_health: HEALTHY } } }
W20260620 07:13:25.133589 22857 tablet.cc:2369] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:13:25.135178 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:25.129782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40312 (local address 127.6.53.66:45665)
0620 07:13:25.129968 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:25.129973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:25.129990 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:13:25.130163 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:25.130166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:25.130243 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:25.130407 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:13:25.130744 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:25.131476 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:25.132807 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:25.133191 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:25.133286 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:25.133783 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:13:25.134544 (+ 761us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:13:25.134664 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:25.134769 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 07:13:25.134869 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:13:25.142868 22548 raft_consensus.cc:1261] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Refusing update from remote peer a22e4f123f9046bdae9441e9f5146271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:13:25.142874 22811 raft_consensus.cc:1261] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9 [term 1 FOLLOWER]: Refusing update from remote peer a22e4f123f9046bdae9441e9f5146271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:13:25.143878 22883 consensus_queue.cc:1035] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:13:25.144258 22879 consensus_queue.cc:1035] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:13:25.154634 22885 mvcc.cc:205] Tried to move back new op lower bound from 7298824622655156224 to 7298824622507573248. Current Snapshot: MvccSnapshot[applied={T|T < 7298824622655156224}]
I20260620 07:13:25.161499 22886 mvcc.cc:205] Tried to move back new op lower bound from 7298824622655156224 to 7298824622507573248. Current Snapshot: MvccSnapshot[applied={T|T < 7298824622655156224}]
I20260620 07:13:25.162467 22887 mvcc.cc:205] Tried to move back new op lower bound from 7298824622655156224 to 7298824622507573248. Current Snapshot: MvccSnapshot[applied={T|T < 7298824622655156224}]
I20260620 07:13:25.169574 22408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:13:25.169836 22408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:13:25.173321 22408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Sending DeleteTablet for 3 replicas of tablet 6c9223f8390c48618b175416318579f6
I20260620 07:13:25.174434 22528 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9223f8390c48618b175416318579f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:13:25 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:48684
I20260620 07:13:25.174433 22791 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9223f8390c48618b175416318579f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:13:25 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:51128
I20260620 07:13:25.174746 22660 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c9223f8390c48618b175416318579f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:13:25 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:40308
I20260620 07:13:25.175745 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 22445
I20260620 07:13:25.208374 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:25.177574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.6.53.65:42971)
0620 07:13:25.177799 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:25.177807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:25.177828 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:13:25.177892 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:25.177900 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:25.177993 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:25.178123 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:25.180543 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:25.182128 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:25.184747 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:25.185703 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:25.185858 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:25.194809 (+ 8951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:25.194829 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:25.194848 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:25.194887 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:25.200629 (+ 5742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:25.201947 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:25.201961 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:25.201967 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:25.202165 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:25.203927 (+ 1762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:25.203936 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:25.203942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:25.204610 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:25.204974 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:25.207252 (+ 2278us) server_negotiation.cc:299] Negotiation successful
0620 07:13:25.207726 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:13:25.210996 22588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:13:25.214752 22897 tablet_replica.cc:330] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9: stopping tablet replica
I20260620 07:13:25.215492 22898 tablet_replica.cc:330] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271: stopping tablet replica
I20260620 07:13:25.216308 22898 raft_consensus.cc:2229] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:13:25.217001 22897 raft_consensus.cc:2229] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:13:25.217759 22897 raft_consensus.cc:2258] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:13:25.218680 22898 raft_consensus.cc:2258] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:13:25.221966 22897 ts_tablet_manager.cc:1905] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:13:25.224644 22898 ts_tablet_manager.cc:1905] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:13:25.235204 22897 ts_tablet_manager.cc:1918] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:13:25.235394 22897 log.cc:1196] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/6c9223f8390c48618b175416318579f6
I20260620 07:13:25.235934 22897 ts_tablet_manager.cc:1939] T 6c9223f8390c48618b175416318579f6 P ea601c5dde2949a5b97dab19cacd2bf9: Deleting consensus metadata
I20260620 07:13:25.237895 22395 catalog_manager.cc:4841] TS ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853): tablet 6c9223f8390c48618b175416318579f6 (table test-table [id=3219ce40af444a4ba97a19b76b373c78]) successfully deleted
I20260620 07:13:25.242862 22898 ts_tablet_manager.cc:1918] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:13:25.243042 22898 log.cc:1196] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/6c9223f8390c48618b175416318579f6
I20260620 07:13:25.243657 22898 ts_tablet_manager.cc:1939] T 6c9223f8390c48618b175416318579f6 P a22e4f123f9046bdae9441e9f5146271: Deleting consensus metadata
I20260620 07:13:25.245600 22393 catalog_manager.cc:4841] TS a22e4f123f9046bdae9441e9f5146271 (127.6.53.66:45665): tablet 6c9223f8390c48618b175416318579f6 (table test-table [id=3219ce40af444a4ba97a19b76b373c78]) successfully deleted
I20260620 07:13:25.327172 22896 tablet_replica.cc:330] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8: stopping tablet replica
I20260620 07:13:25.327564 22896 raft_consensus.cc:2229] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:13:25.328114 22896 raft_consensus.cc:2258] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:13:25.329319 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 22445
I20260620 07:13:25.346982 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 22597
W20260620 07:13:25.346997 22392 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.53.65:42971 (error 108)
I20260620 07:13:25.349318 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:25.348914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42971 (local address 127.0.0.1:48692)
0620 07:13:25.349078 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:25.349184 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 07:13:25.349633 22392 catalog_manager.cc:4584] TS cedbe51127774d0fb853d8da4b172ac8 (127.6.53.65:42971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c9223f8390c48618b175416318579f6: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42971: connect: Connection refused (error 111)
I20260620 07:13:25.367937 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:25.348187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.6.53.66:45665)
0620 07:13:25.348384 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:25.348388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:25.348405 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:13:25.348459 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:25.348463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:25.348519 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:25.348604 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:25.350457 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:25.351295 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:25.353239 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:25.353662 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:25.353775 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:25.361905 (+ 8130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:25.361928 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:25.361944 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:25.361982 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:25.365110 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:25.365959 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:25.365966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:25.365968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:25.366108 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:25.366524 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:25.366527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:25.366529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:25.366883 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:25.367147 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:25.367459 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 07:13:25.367621 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:13:25.368695 22720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:13:25.403136 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:25.402636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42971 (local address 127.0.0.1:48706)
0620 07:13:25.402844 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:25.402963 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 07:13:25.404165 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:25.403789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42971 (local address 127.0.0.1:48716)
0620 07:13:25.403940 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:25.404021 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 07:13:25.464819 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:25.464195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42971 (local address 127.0.0.1:48720)
0620 07:13:25.464438 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:25.464551 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 07:13:25.465987 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:25.465593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42971 (local address 127.0.0.1:48722)
0620 07:13:25.465772 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:25.465855 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 07:13:25.483567 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 22597
I20260620 07:13:25.502164 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 22728
I20260620 07:13:25.515137 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:25.502938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51130 (local address 127.6.53.67:35853)
0620 07:13:25.503117 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:25.503125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:25.503147 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:13:25.503371 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:25.503376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:25.503461 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:25.503570 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:25.504818 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:25.505538 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:25.506863 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:25.507264 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:25.507404 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:25.509805 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:25.509818 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:25.509833 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:25.509863 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:25.512412 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:25.513082 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:25.513091 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:25.513095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:25.513234 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:25.513596 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:25.513602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:25.513606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:25.513943 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:25.514239 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:25.514588 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 07:13:25.514822 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:13:25.515797 22851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:13:25.547312 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:25.546745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42971 (local address 127.0.0.1:48736)
0620 07:13:25.546971 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:25.547086 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 07:13:25.548373 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:25.547959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:42971 (local address 127.0.0.1:48748)
0620 07:13:25.548159 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:25.548254 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:42971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 07:13:25.604734 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 22728
I20260620 07:13:25.621037 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 22375
I20260620 07:13:25.622145 22438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:13:25.733487 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 22375
I20260620 07:13:45.754980 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:41535
--webserver_interface=127.6.53.126
--webserver_port=45807
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:41535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:13:46.078696 22904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:13:46.078950 22904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:13:46.079119 22904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:13:46.086218 22904 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 07:13:46.086390 22904 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 07:13:46.086454 22904 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 07:13:46.086532 22904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:13:46.086620 22904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:13:46.086736 22904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:13:46.086818 22904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:13:46.086900 22904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:13:46.096427 22904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:41535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:41535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=45807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:13:46.097668 22904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:13:46.098989 22904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:13:46.110983 22909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:46.111089 22910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:46.111050 22912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:46.111513 22904 server_base.cc:1034] running on GCE node
I20260620 07:13:46.112315 22904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:13:46.113555 22904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:13:46.114759 22904 hybrid_clock.cc:648] HybridClock initialized: now 1781939626114699 us; error 42 us; skew 500 ppm
I20260620 07:13:46.115064 22904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:13:46 dist-test-slave-4n6h krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939626, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:13:46.118067 22904 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:13:46.119411 22904 webserver.cc:460] Webserver started at http://127.6.53.126:45807/ using document root <none> and password file <none>
I20260620 07:13:46.120064 22904 fs_manager.cc:359] Metadata directory not provided
I20260620 07:13:46.120174 22904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:13:46.125169 22904 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 07:13:46.127794 22919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:46.128957 22904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:13:46.129125 22904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "b5505e04949e465d97a722d359ce9ec8"
format_stamp: "Formatted at 2026-06-20 07:13:00 on dist-test-slave-4n6h"
server_key: "afc4a5dcf67c6da4a35da16adac3aa55"
server_key_iv: "49994c063214bf95477b09c69a4d56b6"
server_key_version: "encryptionkey@0"
I20260620 07:13:46.130122 22904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:13:46.146350 22904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:13:46.149983 22904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:13:46.150282 22904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:13:46.167812 22904 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:41535
I20260620 07:13:46.167860 22970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:41535 every 8 connection(s)
I20260620 07:13:46.169195 22904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:13:46.175650 22971 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:13:46.176679 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 22904
I20260620 07:13:46.177712 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:42971
--local_ip_for_outbound_sockets=127.6.53.65
--tserver_master_addrs=127.6.53.126:41535
--webserver_port=41457
--webserver_interface=127.6.53.65
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:13:46.180390 22971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Bootstrap starting.
I20260620 07:13:46.202342 22971 log.cc:824] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:13:46.217877 22971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:13:46.218473 22971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Bootstrap complete.
I20260620 07:13:46.227766 22971 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } }
I20260620 07:13:46.228579 22971 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5505e04949e465d97a722d359ce9ec8, State: Initialized, Role: FOLLOWER
I20260620 07:13:46.229194 22971 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } }
I20260620 07:13:46.229446 22971 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:13:46.229602 22971 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:13:46.229779 22971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 2 FOLLOWER]: Advancing to term 3
I20260620 07:13:46.233532 22971 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } }
I20260620 07:13:46.234025 22971 leader_election.cc:302] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5505e04949e465d97a722d359ce9ec8; no voters:
I20260620 07:13:46.234714 22971 leader_election.cc:288] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 07:13:46.234900 22975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 07:13:46.236012 22975 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [term 3 LEADER]: Becoming Leader. State: Replica: b5505e04949e465d97a722d359ce9ec8, State: Running, Role: LEADER
I20260620 07:13:46.236693 22971 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:13:46.236609 22975 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } }
I20260620 07:13:46.242827 22976 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b5505e04949e465d97a722d359ce9ec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } } }
I20260620 07:13:46.243146 22976 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:13:46.243767 22977 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5505e04949e465d97a722d359ce9ec8. Latest consensus state: current_term: 3 leader_uuid: "b5505e04949e465d97a722d359ce9ec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5505e04949e465d97a722d359ce9ec8" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41535 } } }
I20260620 07:13:46.244124 22977 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:13:46.252775 22991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:13:46.261596 22991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:13:46.262989 22991 catalog_manager.cc:1259] Loaded cluster ID: 4f5c2c9414ba41938cabda84525d68f2
I20260620 07:13:46.263082 22991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:13:46.266992 22991 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:13:46.268543 22991 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Loaded TSK: 0
I20260620 07:13:46.268679 22991 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b5505e04949e465d97a722d359ce9ec8: Loaded TSK: 1
F20260620 07:13:46.269749 22991 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f71c815a9ad google::LogMessage::Fail() at ??:0
@ 0x7f71c815d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f71c815a4de google::LogMessage::Flush() at ??:0
@ 0x7f71c815ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f71cbebe7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f71cbead73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f71ddb891c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f71ddb88783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f71ddc142a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f71ddc140c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f71ddc5e5e9 std::function<>::operator()() at ??:0
@ 0x7f71ddb8ccdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f71ddb8ac36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f71ddc12610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f71ddc12439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f71cada6345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f71cada991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f71cada9799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f71cad84eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f71ce05e6db start_thread at ??:0
@ 0x7f71c6ff461f clone at ??:0
W20260620 07:13:46.562749 22973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:13:46.563031 22973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:13:46.563186 22973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:13:46.570484 22973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:13:46.570586 22973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:13:46.570807 22973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:13:46.580958 22973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:42971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=41457
--tserver_master_addrs=127.6.53.126:41535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:13:46.582343 22973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:13:46.583702 22973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:13:46.597333 23018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:46.597275 23021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:46.597288 23019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:46.598204 22973 server_base.cc:1034] running on GCE node
I20260620 07:13:46.598855 22973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:13:46.599792 22973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:13:46.600983 22973 hybrid_clock.cc:648] HybridClock initialized: now 1781939626600949 us; error 36 us; skew 500 ppm
I20260620 07:13:46.601303 22973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:13:46 dist-test-slave-4n6h krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939626, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:13:46.604422 22973 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:13:46.605911 22973 webserver.cc:460] Webserver started at http://127.6.53.65:41457/ using document root <none> and password file <none>
I20260620 07:13:46.606683 22973 fs_manager.cc:359] Metadata directory not provided
I20260620 07:13:46.606812 22973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:13:46.611922 22973 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:13:46.614584 23028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:46.615754 22973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:13:46.615913 22973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "cedbe51127774d0fb853d8da4b172ac8"
format_stamp: "Formatted at 2026-06-20 07:13:00 on dist-test-slave-4n6h"
server_key: "8d36984950388f405d3b7badb6e59ad6"
server_key_iv: "0ee35a25fdcda511ccabc85c6b6c6059"
server_key_version: "encryptionkey@0"
I20260620 07:13:46.616850 22973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:13:46.643429 22973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:13:46.647313 22973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:13:46.647588 22973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:13:46.648535 22973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:13:46.650231 23035 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 07:13:46.655843 22973 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 07:13:46.656003 22973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 07:13:46.656144 22973 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 07:13:46.659179 22973 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 07:13:46.659282 22973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260620 07:13:46.659587 23035 tablet_bootstrap.cc:492] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8: Bootstrap starting.
I20260620 07:13:46.667927 23035 log.cc:824] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:13:46.682838 23035 tablet_bootstrap.cc:492] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:13:46.683799 23035 tablet_bootstrap.cc:492] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8: Bootstrap complete.
I20260620 07:13:46.684394 23035 ts_tablet_manager.cc:1397] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.005s
I20260620 07:13:46.695137 23035 raft_consensus.cc:348] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:46.696679 23035 raft_consensus.cc:729] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cedbe51127774d0fb853d8da4b172ac8, State: Initialized, Role: FOLLOWER
I20260620 07:13:46.697484 23035 consensus_queue.cc:260] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:46.699179 23035 ts_tablet_manager.cc:1428] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260620 07:13:46.700492 22973 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:42971
I20260620 07:13:46.700515 23143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:42971 every 8 connection(s)
I20260620 07:13:46.701704 22973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:13:46.704726 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 22973
I20260620 07:13:46.705356 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:45665
--local_ip_for_outbound_sockets=127.6.53.66
--tserver_master_addrs=127.6.53.126:41535
--webserver_port=33949
--webserver_interface=127.6.53.66
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:13:46.706668 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:46.704412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:41635)
0620 07:13:46.705332 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:46.705616 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":421,"threads_started":1}
W20260620 07:13:46.707376 23144 heartbeater.cc:644] Failed to heartbeat to 127.6.53.126:41535 (0 consecutive failures): Network error: Failed to ping master at 127.6.53.126:41535: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
I20260620 07:13:46.708480 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:46.707952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:49113)
0620 07:13:46.708184 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:46.708267 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260620 07:13:46.709631 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:46.709142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:35205)
0620 07:13:46.709368 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:46.709456 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260620 07:13:46.709961 23144 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:13:47.035187 23147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:13:47.035432 23147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:13:47.035625 23147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:13:47.042798 23147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:13:47.042917 23147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:13:47.043098 23147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:13:47.052417 23147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:45665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=33949
--tserver_master_addrs=127.6.53.126:41535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:13:47.053594 23147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:13:47.054883 23147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:13:47.067696 23152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:47.067734 23155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:47.067659 23153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:47.068774 23147 server_base.cc:1034] running on GCE node
I20260620 07:13:47.069365 23147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:13:47.070205 23147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:13:47.071441 23147 hybrid_clock.cc:648] HybridClock initialized: now 1781939627071391 us; error 46 us; skew 500 ppm
I20260620 07:13:47.071725 23147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:13:47 dist-test-slave-4n6h krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939627, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:13:47.074677 23147 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:13:47.076004 23147 webserver.cc:460] Webserver started at http://127.6.53.66:33949/ using document root <none> and password file <none>
I20260620 07:13:47.076572 23147 fs_manager.cc:359] Metadata directory not provided
I20260620 07:13:47.076670 23147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:13:47.081679 23147 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:13:47.085301 23162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:47.086732 23147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:13:47.086920 23147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "a22e4f123f9046bdae9441e9f5146271"
format_stamp: "Formatted at 2026-06-20 07:13:01 on dist-test-slave-4n6h"
server_key: "051ccf6afa29750a9515ac12ee1fb1af"
server_key_iv: "0c82811afd3444bf6f1dc3bd35389644"
server_key_version: "encryptionkey@0"
I20260620 07:13:47.087920 23147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:13:47.108285 23147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:13:47.112119 23147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:13:47.112390 23147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:13:47.113332 23147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:13:47.114868 23147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:13:47.114962 23147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:47.115118 23147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:13:47.115200 23147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:47.148013 23147 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:45665
I20260620 07:13:47.148097 23274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:45665 every 8 connection(s)
I20260620 07:13:47.149454 23147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:13:47.153805 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:47.152160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:39085)
0620 07:13:47.152834 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:47.153027 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":272,"threads_started":1}
W20260620 07:13:47.154511 23275 heartbeater.cc:644] Failed to heartbeat to 127.6.53.126:41535 (0 consecutive failures): Network error: Failed to ping master at 127.6.53.126:41535: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
I20260620 07:13:47.155267 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:47.154981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:55369)
0620 07:13:47.155110 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:47.155172 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 07:13:47.156098 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:47.155754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:58247)
0620 07:13:47.155901 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:47.156000 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 07:13:47.156374 23275 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:13:47.158764 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 23147
I20260620 07:13:47.159379 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:35853
--local_ip_for_outbound_sockets=127.6.53.67
--tserver_master_addrs=127.6.53.126:41535
--webserver_port=42879
--webserver_interface=127.6.53.67
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:13:47.475623 23278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:13:47.475869 23278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:13:47.476029 23278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:13:47.483057 23278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:13:47.483155 23278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:13:47.483359 23278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:13:47.493028 23278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:35853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=42879
--tserver_master_addrs=127.6.53.126:41535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:13:47.494230 23278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:13:47.495579 23278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:13:47.508080 23283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:47.508111 23286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:47.508078 23284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:47.508940 23278 server_base.cc:1034] running on GCE node
I20260620 07:13:47.509538 23278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:13:47.510465 23278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:13:47.511687 23278 hybrid_clock.cc:648] HybridClock initialized: now 1781939627511654 us; error 45 us; skew 500 ppm
I20260620 07:13:47.511981 23278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:13:47 dist-test-slave-4n6h krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939627, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:13:47.515127 23278 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:13:47.516551 23278 webserver.cc:460] Webserver started at http://127.6.53.67:42879/ using document root <none> and password file <none>
I20260620 07:13:47.517180 23278 fs_manager.cc:359] Metadata directory not provided
I20260620 07:13:47.517328 23278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:13:47.522622 23278 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:13:47.525121 23293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:47.526383 23278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:13:47.526552 23278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "ea601c5dde2949a5b97dab19cacd2bf9"
format_stamp: "Formatted at 2026-06-20 07:13:01 on dist-test-slave-4n6h"
server_key: "4c62ea3562631c705609db1eea49edd9"
server_key_iv: "5b2c67f4dcfdcf833631bcce617a2937"
server_key_version: "encryptionkey@0"
I20260620 07:13:47.527521 23278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:13:47.545161 23278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:13:47.549497 23278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:13:47.549778 23278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:13:47.550807 23278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:13:47.552088 23278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:13:47.552208 23278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:47.552347 23278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:13:47.552438 23278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:47.586175 23278 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:35853
I20260620 07:13:47.586272 23405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:35853 every 8 connection(s)
I20260620 07:13:47.587597 23278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:13:47.592716 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:47.590551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:52125)
0620 07:13:47.591481 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:47.591690 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":411,"threads_started":1}
I20260620 07:13:47.592964 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 23278
W20260620 07:13:47.593523 23406 heartbeater.cc:644] Failed to heartbeat to 127.6.53.126:41535 (0 consecutive failures): Network error: Failed to ping master at 127.6.53.126:41535: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
I20260620 07:13:47.594512 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:47.594031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:33375)
0620 07:13:47.594231 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:47.594370 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 07:13:47.595440 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:47.595094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:41451)
0620 07:13:47.595244 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:47.595330 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 07:13:47.595681 23406 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:13:47.711129 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:47.710468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:39209)
0620 07:13:47.710864 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:47.710935 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":187,"threads_started":1}
I20260620 07:13:47.861049 23411 raft_consensus.cc:482] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:13:47.861279 23411 raft_consensus.cc:504] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:47.862972 23411 leader_election.cc:288] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853), a22e4f123f9046bdae9441e9f5146271 (127.6.53.66:45665)
Jun 20 07:13:47 dist-test-slave-4n6h krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939626, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
Jun 20 07:13:47 dist-test-slave-4n6h krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939626, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:13:47.885285 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:47.863761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:49049 (local address 127.6.53.67:35853)
0620 07:13:47.865398 (+ 1637us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:47.865411 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:47.865457 (+ 46us) server_negotiation.cc:407] Connection header received
0620 07:13:47.867713 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:47.867727 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:47.868187 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:47.868537 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:47.870592 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:47.871748 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:47.873442 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:47.874026 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:47.874242 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:47.878166 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:47.878173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:47.878182 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:47.878218 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:47.881629 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:47.882520 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:47.882526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:47.882531 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:47.882680 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:47.883099 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:47.883109 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:47.883111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:47.883572 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:47.883769 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:47.884505 (+ 736us) server_negotiation.cc:299] Negotiation successful
0620 07:13:47.884875 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1427,"thread_start_us":1237,"threads_started":1}
I20260620 07:13:47.885290 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:47.863547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:51181 (local address 127.6.53.66:45665)
0620 07:13:47.865569 (+ 2022us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:47.865583 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:47.865614 (+ 31us) server_negotiation.cc:407] Connection header received
0620 07:13:47.867740 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:47.867749 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:47.868187 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:47.868537 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:47.870583 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:47.871718 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:47.873401 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:47.874028 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:47.874223 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:47.877910 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:47.877920 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:47.877938 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:47.877986 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:47.881629 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:47.882511 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:47.882518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:47.882522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:47.882669 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:47.883105 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:47.883109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:47.883111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:47.883573 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:47.883767 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:47.884502 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 07:13:47.884873 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1871,"thread_start_us":1760,"threads_started":1}
I20260620 07:13:47.885318 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:47.863285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:45665 (local address 127.6.53.65:51181)
0620 07:13:47.864975 (+ 1690us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:47.865008 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:47.865133 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:47.868690 (+ 3557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:47.868701 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:47.869608 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:13:47.870377 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:47.870397 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:47.871915 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:47.871925 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:47.873263 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:47.873274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:47.874322 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:47.874330 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:47.874531 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:47.875274 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:13:47.875302 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:13:47.877616 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:13:47.881810 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:47.881815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:47.881819 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:47.882361 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:47.882783 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:47.882789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:47.882793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:47.882975 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:47.883705 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:13:47.883712 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:13:47.884025 (+ 313us) client_negotiation.cc:769] Sending connection context
0620 07:13:47.884277 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 07:13:47.884831 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1530,"mutex_wait_us":137,"spinlock_wait_cycles":18176,"thread_start_us":1274,"threads_started":1}
I20260620 07:13:47.885670 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:47.863159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:35853 (local address 127.6.53.65:49049)
0620 07:13:47.863506 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:47.863543 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:47.864132 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:47.868690 (+ 4558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:47.868701 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:47.869649 (+ 948us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:13:47.870390 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:47.870404 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:47.871931 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:47.871934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:47.873269 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:47.873281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:47.874241 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:47.874271 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:47.874531 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:47.875347 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:13:47.875364 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:13:47.877955 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:13:47.881775 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:47.881781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:47.881792 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:47.882376 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:47.882804 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:47.882809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:47.882813 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:47.882979 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:47.883707 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:13:47.883714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:13:47.884025 (+ 311us) client_negotiation.cc:769] Sending connection context
0620 07:13:47.884277 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 07:13:47.884981 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":387,"spinlock_wait_cycles":33920}
W20260620 07:13:47.888350 23030 leader_election.cc:341] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a22e4f123f9046bdae9441e9f5146271 (127.6.53.66:45665): Not found: Tablet not found: 6c9223f8390c48618b175416318579f6
W20260620 07:13:47.888569 23032 leader_election.cc:341] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853): Not found: Tablet not found: 6c9223f8390c48618b175416318579f6
I20260620 07:13:47.888731 23032 leader_election.cc:302] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cedbe51127774d0fb853d8da4b172ac8; no voters: a22e4f123f9046bdae9441e9f5146271, ea601c5dde2949a5b97dab19cacd2bf9
I20260620 07:13:47.889137 23411 raft_consensus.cc:2726] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:13:48.157701 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:48.156904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:49949)
0620 07:13:48.157347 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:48.157479 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":222,"threads_started":1}
I20260620 07:13:48.596825 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:48.596149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:57849)
0620 07:13:48.596534 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:48.596634 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":193,"threads_started":1}
I20260620 07:13:48.712754 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:48.712046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:35429)
0620 07:13:48.712521 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:48.712614 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":289,"threads_started":1}
I20260620 07:13:49.159243 23418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:49.158551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:43801)
0620 07:13:49.159028 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:49.159099 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":283,"threads_started":1}
I20260620 07:13:49.598584 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:49.597656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:51051)
0620 07:13:49.598227 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:49.598397 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":295,"threads_started":1}
I20260620 07:13:49.714422 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:49.713611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:38655)
0620 07:13:49.714061 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:49.714145 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":218,"threads_started":1}
I20260620 07:13:49.996510 23421 raft_consensus.cc:482] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:13:49.996717 23421 raft_consensus.cc:504] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:49.997411 23421 leader_election.cc:288] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853), a22e4f123f9046bdae9441e9f5146271 (127.6.53.66:45665)
W20260620 07:13:49.998337 23032 leader_election.cc:341] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853): Not found: Tablet not found: 6c9223f8390c48618b175416318579f6
W20260620 07:13:49.998589 23030 leader_election.cc:341] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a22e4f123f9046bdae9441e9f5146271 (127.6.53.66:45665): Not found: Tablet not found: 6c9223f8390c48618b175416318579f6
I20260620 07:13:49.998761 23030 leader_election.cc:302] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cedbe51127774d0fb853d8da4b172ac8; no voters: a22e4f123f9046bdae9441e9f5146271, ea601c5dde2949a5b97dab19cacd2bf9
I20260620 07:13:49.999046 23421 raft_consensus.cc:2726] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:13:50.160832 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:50.160049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:51271)
0620 07:13:50.160587 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:50.160674 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":321,"threads_started":1}
I20260620 07:13:50.600080 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:50.599441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:39309)
0620 07:13:50.599832 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:50.599914 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":207,"threads_started":1}
I20260620 07:13:50.715843 23424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:50.715208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:54945)
0620 07:13:50.715611 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:50.715683 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":208,"threads_started":1}
I20260620 07:13:51.162664 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:51.161799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:58923)
0620 07:13:51.162363 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:51.162465 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":237,"threads_started":1}
I20260620 07:13:51.601862 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:51.601084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:44309)
0620 07:13:51.601584 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:51.601675 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":257,"threads_started":1}
I20260620 07:13:51.717710 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:51.716961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:41787)
0620 07:13:51.717441 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:51.717543 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":242,"threads_started":1}
I20260620 07:13:52.005282 23428 raft_consensus.cc:482] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:13:52.005529 23428 raft_consensus.cc:504] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:52.006421 23428 leader_election.cc:288] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853), a22e4f123f9046bdae9441e9f5146271 (127.6.53.66:45665)
W20260620 07:13:52.007601 23030 leader_election.cc:341] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a22e4f123f9046bdae9441e9f5146271 (127.6.53.66:45665): Not found: Tablet not found: 6c9223f8390c48618b175416318579f6
W20260620 07:13:52.007805 23032 leader_election.cc:341] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853): Not found: Tablet not found: 6c9223f8390c48618b175416318579f6
I20260620 07:13:52.007974 23032 leader_election.cc:302] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cedbe51127774d0fb853d8da4b172ac8; no voters: a22e4f123f9046bdae9441e9f5146271, ea601c5dde2949a5b97dab19cacd2bf9
I20260620 07:13:52.008245 23428 raft_consensus.cc:2726] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:13:52.164300 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:52.163535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:53251)
0620 07:13:52.164041 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:52.164121 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":287,"threads_started":1}
I20260620 07:13:52.603711 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:52.602957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:36219)
0620 07:13:52.603475 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:52.603562 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":280,"threads_started":1}
I20260620 07:13:52.719271 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:52.718618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:46451)
0620 07:13:52.719036 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:52.719119 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":209,"threads_started":1}
I20260620 07:13:53.165891 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:53.165165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:49265)
0620 07:13:53.165620 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:53.165701 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":243,"threads_started":1}
I20260620 07:13:53.605198 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:53.604509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:35179)
0620 07:13:53.604960 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:53.605034 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":239,"threads_started":1}
I20260620 07:13:53.720983 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:53.720171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:42901)
0620 07:13:53.720655 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:53.720754 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":220,"threads_started":1}
I20260620 07:13:54.167460 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:54.166723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:58315)
0620 07:13:54.167181 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:54.167252 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":216,"threads_started":1}
I20260620 07:13:54.606822 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:54.606027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:36353)
0620 07:13:54.606544 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:54.606620 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":279,"threads_started":1}
I20260620 07:13:54.722469 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:54.721779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:46805)
0620 07:13:54.722178 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:54.722317 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":199,"threads_started":1}
I20260620 07:13:55.169278 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:55.168442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:34933)
0620 07:13:55.168994 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:55.169079 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":314,"threads_started":1}
I20260620 07:13:55.608381 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:55.607704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:39101)
0620 07:13:55.608127 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:55.608204 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":240,"threads_started":1}
I20260620 07:13:55.724283 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:55.723326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:37421)
0620 07:13:55.723976 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:55.724075 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":364,"threads_started":1}
I20260620 07:13:56.171077 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:56.170334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:49049)
0620 07:13:56.170784 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:56.170859 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":203,"threads_started":1}
I20260620 07:13:56.610077 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:56.609248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:52883)
0620 07:13:56.609791 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:56.609894 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":267,"threads_started":1}
I20260620 07:13:56.725870 23443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:56.725222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:57261)
0620 07:13:56.725643 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:56.725719 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":224,"threads_started":1}
I20260620 07:13:57.052644 23444 raft_consensus.cc:482] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:13:57.052865 23444 raft_consensus.cc:504] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:57.053524 23444 leader_election.cc:288] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853), a22e4f123f9046bdae9441e9f5146271 (127.6.53.66:45665)
W20260620 07:13:57.054533 23030 leader_election.cc:341] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a22e4f123f9046bdae9441e9f5146271 (127.6.53.66:45665): Not found: Tablet not found: 6c9223f8390c48618b175416318579f6
W20260620 07:13:57.054795 23032 leader_election.cc:341] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853): Not found: Tablet not found: 6c9223f8390c48618b175416318579f6
I20260620 07:13:57.054901 23032 leader_election.cc:302] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cedbe51127774d0fb853d8da4b172ac8; no voters: a22e4f123f9046bdae9441e9f5146271, ea601c5dde2949a5b97dab19cacd2bf9
I20260620 07:13:57.055105 23444 raft_consensus.cc:2726] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:13:57.172787 23445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:57.172064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:44971)
0620 07:13:57.172512 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:57.172617 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":258,"threads_started":1}
I20260620 07:13:57.611683 23446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:57.610986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:53087)
0620 07:13:57.611443 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:57.611520 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":246,"threads_started":1}
I20260620 07:13:57.727365 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:57.726706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:52561)
0620 07:13:57.727116 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:57.727205 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":221,"threads_started":1}
I20260620 07:13:58.174440 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:58.173678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:47917)
0620 07:13:58.174129 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:58.174214 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":215,"threads_started":1}
I20260620 07:13:58.613265 23449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:58.612546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:33473)
0620 07:13:58.613035 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:58.613110 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":263,"threads_started":1}
I20260620 07:13:58.728801 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:58.728145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:46965)
0620 07:13:58.728561 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:58.728636 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":235,"threads_started":1}
I20260620 07:13:59.176036 23451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:59.175379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:43331)
0620 07:13:59.175796 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:59.175877 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":226,"threads_started":1}
I20260620 07:13:59.248651 23452 raft_consensus.cc:482] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:13:59.248867 23452 raft_consensus.cc:504] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:13:59.249536 23452 leader_election.cc:288] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853), a22e4f123f9046bdae9441e9f5146271 (127.6.53.66:45665)
W20260620 07:13:59.250485 23030 leader_election.cc:341] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a22e4f123f9046bdae9441e9f5146271 (127.6.53.66:45665): Not found: Tablet not found: 6c9223f8390c48618b175416318579f6
W20260620 07:13:59.250614 23032 leader_election.cc:341] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853): Not found: Tablet not found: 6c9223f8390c48618b175416318579f6
I20260620 07:13:59.250814 23032 leader_election.cc:302] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cedbe51127774d0fb853d8da4b172ac8; no voters: a22e4f123f9046bdae9441e9f5146271, ea601c5dde2949a5b97dab19cacd2bf9
I20260620 07:13:59.251109 23452 raft_consensus.cc:2726] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:13:59.614979 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:59.614216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:45945)
0620 07:13:59.614724 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:59.614800 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":254,"threads_started":1}
I20260620 07:13:59.730816 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:59.729675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:51223)
0620 07:13:59.730371 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:59.730512 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":320,"threads_started":1}
I20260620 07:14:00.177861 23455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:00.177129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:59371)
0620 07:14:00.177593 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:00.177677 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":250,"threads_started":1}
I20260620 07:14:00.616612 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:00.615902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:49435)
0620 07:14:00.616364 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:00.616452 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":250,"threads_started":1}
I20260620 07:14:00.732535 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:00.731759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:59841)
0620 07:14:00.732270 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:00.732346 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":290,"threads_started":1}
I20260620 07:14:01.179553 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:01.178807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:50359)
0620 07:14:01.179262 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:01.179363 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":248,"threads_started":1}
I20260620 07:14:01.377828 23459 raft_consensus.cc:482] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:01.378050 23459 raft_consensus.cc:504] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cedbe51127774d0fb853d8da4b172ac8" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 42971 } } peers { permanent_uuid: "ea601c5dde2949a5b97dab19cacd2bf9" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 35853 } } peers { permanent_uuid: "a22e4f123f9046bdae9441e9f5146271" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45665 } }
I20260620 07:14:01.378710 23459 leader_election.cc:288] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853), a22e4f123f9046bdae9441e9f5146271 (127.6.53.66:45665)
W20260620 07:14:01.379812 23032 leader_election.cc:341] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea601c5dde2949a5b97dab19cacd2bf9 (127.6.53.67:35853): Not found: Tablet not found: 6c9223f8390c48618b175416318579f6
W20260620 07:14:01.380008 23030 leader_election.cc:341] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a22e4f123f9046bdae9441e9f5146271 (127.6.53.66:45665): Not found: Tablet not found: 6c9223f8390c48618b175416318579f6
I20260620 07:14:01.380097 23030 leader_election.cc:302] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cedbe51127774d0fb853d8da4b172ac8; no voters: a22e4f123f9046bdae9441e9f5146271, ea601c5dde2949a5b97dab19cacd2bf9
I20260620 07:14:01.380343 23459 raft_consensus.cc:2726] T 6c9223f8390c48618b175416318579f6 P cedbe51127774d0fb853d8da4b172ac8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:14:01.618192 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:01.617482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:51253)
0620 07:14:01.617929 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:01.618008 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":232,"threads_started":1}
I20260620 07:14:01.734549 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:01.733610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.65:60941)
0620 07:14:01.734084 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:01.734162 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":214,"threads_started":1}
I20260620 07:14:02.181399 23462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:02.180678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.66:51741)
0620 07:14:02.181133 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:02.181249 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":219,"threads_started":1}
I20260620 07:14:02.602473 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 22973
I20260620 07:14:02.620128 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:02.619241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41535 (local address 127.6.53.67:48383)
0620 07:14:02.619750 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:02.619921 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.126:41535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":276,"threads_started":1}
I20260620 07:14:02.627243 23463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:02.603543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60862 (local address 127.6.53.65:42971)
0620 07:14:02.604153 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:02.604160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:02.604217 (+ 57us) server_negotiation.cc:407] Connection header received
0620 07:14:02.604633 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:02.604646 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:02.604910 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:02.605080 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:02.606552 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:02.607502 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:02.608623 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:02.609149 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:02.609242 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:02.621931 (+ 12689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:02.621942 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:02.621955 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:02.621987 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:02.624892 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:02.625497 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:02.625503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:02.625506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:02.625627 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:02.625938 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:02.625942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:02.625943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:02.626341 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:02.626556 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:02.626795 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 07:14:02.626992 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":404,"threads_started":1}
I20260620 07:14:02.628400 23138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:02.716795 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 22973
I20260620 07:14:02.734874 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 23147
I20260620 07:14:02.747488 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:02.735601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51336 (local address 127.6.53.66:45665)
0620 07:14:02.736130 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:02.736134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:02.736150 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:14:02.736206 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:02.736210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:02.736266 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:02.736357 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:02.737675 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:02.738530 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:02.739686 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:02.740097 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:02.740186 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:02.742704 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:02.742708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:02.742711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:02.742742 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:02.745192 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:02.745782 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:02.745785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:02.745787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:02.745913 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:02.746206 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:02.746209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:02.746211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:02.746503 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:02.746691 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:02.746943 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 07:14:02.747096 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":323,"threads_started":1}
I20260620 07:14:02.748155 23270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:02.831872 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 23147
I20260620 07:14:02.849820 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 23278
I20260620 07:14:02.864174 23468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:02.850761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.6.53.67:35853)
0620 07:14:02.851336 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:02.851341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:02.851357 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:14:02.851410 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:02.851413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:02.851477 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:02.851564 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:02.852914 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:02.853734 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:02.855185 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:02.855628 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:02.855724 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:02.858529 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:02.858534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:02.858537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:02.858573 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:02.861182 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:02.861828 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:02.861833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:02.861837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:02.861970 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:02.862422 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:02.862427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:02.862527 (+ 100us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:02.862812 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:02.863269 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:02.863457 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 07:14:02.863759 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":314,"threads_started":1}
I20260620 07:14:02.864933 23401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:02.948202 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 23278
2026-06-20T07:14:02Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64481 ms)
[----------] 30 tests from SecurityITest (290901 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:03 dist-test-slave-4n6h krb5kdc[23472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:03 dist-test-slave-4n6h krb5kdc[23472](info): set up 2 sockets
Jun 20 07:14:03 dist-test-slave-4n6h krb5kdc[23472](info): commencing operation
krb5kdc: starting...
W20260620 07:14:04.114398 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:04 dist-test-slave-4n6h krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:14:04.349270 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:34553
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:41921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:34553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:04.671955 23485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:04.672250 23485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:04.672343 23485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:04.679531 23485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:04.679644 23485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:04.679690 23485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:04.679749 23485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:04.679792 23485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:04.689595 23485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:41921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:34553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:34553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:04.690937 23485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:04.692173 23485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:04.704073 23493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:04.704159 23490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:04.704135 23491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:04.705440 23485 server_base.cc:1034] running on GCE node
I20260620 07:14:04.706344 23485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:04.707724 23485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:04.708989 23485 hybrid_clock.cc:648] HybridClock initialized: now 1781939644708941 us; error 49 us; skew 500 ppm
I20260620 07:14:04.709439 23485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:04 dist-test-slave-4n6h krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939644, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:04.712735 23485 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:14:04.714211 23485 webserver.cc:460] Webserver started at http://127.6.53.126:33367/ using document root <none> and password file <none>
I20260620 07:14:04.714938 23485 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:04.715085 23485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:04.715411 23485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:04.718487 23485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "c1def444c3b24667aaca2ae35341ea9c"
format_stamp: "Formatted at 2026-06-20 07:14:04 on dist-test-slave-4n6h"
server_key: "eb2f42e58e4dd3084294df9f8c3bd8b6"
server_key_iv: "f3f6ff691eb1f1bbbbce672ac79a2070"
server_key_version: "encryptionkey@0"
I20260620 07:14:04.719250 23485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "c1def444c3b24667aaca2ae35341ea9c"
format_stamp: "Formatted at 2026-06-20 07:14:04 on dist-test-slave-4n6h"
server_key: "eb2f42e58e4dd3084294df9f8c3bd8b6"
server_key_iv: "f3f6ff691eb1f1bbbbce672ac79a2070"
server_key_version: "encryptionkey@0"
I20260620 07:14:04.724406 23485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:14:04.727808 23500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:04.729050 23485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:14:04.729229 23485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "c1def444c3b24667aaca2ae35341ea9c"
format_stamp: "Formatted at 2026-06-20 07:14:04 on dist-test-slave-4n6h"
server_key: "eb2f42e58e4dd3084294df9f8c3bd8b6"
server_key_iv: "f3f6ff691eb1f1bbbbce672ac79a2070"
server_key_version: "encryptionkey@0"
I20260620 07:14:04.729460 23485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:04.750111 23485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:04.754060 23485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:04.754369 23485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:04.772173 23485 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:34553
I20260620 07:14:04.772269 23551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:34553 every 8 connection(s)
I20260620 07:14:04.773509 23485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:04.777119 23552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:04.782172 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 23485
I20260620 07:14:04.782428 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:04.783082 6357 external_mini_cluster.cc:1428] Setting key c10568cfa467f92268bef5b5a611f29c
I20260620 07:14:04.788200 23552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c: Bootstrap starting.
I20260620 07:14:04.791932 23552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:04.793210 23552 log.cc:824] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:04.796597 23552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c: No bootstrap required, opened a new log
Jun 20 07:14:04 dist-test-slave-4n6h krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:14:04.804901 23552 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1def444c3b24667aaca2ae35341ea9c" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 34553 } }
I20260620 07:14:04.805269 23552 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:04.805392 23552 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1def444c3b24667aaca2ae35341ea9c, State: Initialized, Role: FOLLOWER
I20260620 07:14:04.806049 23552 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1def444c3b24667aaca2ae35341ea9c" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 34553 } }
I20260620 07:14:04.806385 23552 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:04.806545 23552 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:04.806736 23552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:04.807433 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:04.785130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53538 (local address 127.6.53.126:34553)
0620 07:14:04.786010 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:04.786022 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:04.786065 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:14:04.787910 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:04.787934 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:04.788321 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:04.788704 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:04.790604 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:04.791868 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:04.793724 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:04.794809 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:04.794935 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:04.799744 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:04.799752 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:04.799766 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:04.799808 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:04.803003 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:04.803664 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:04.803671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:04.803677 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:04.803845 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:04.804252 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:04.804261 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:04.804263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:04.804749 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:04.805044 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:04.805539 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 07:14:04.805951 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":412,"threads_started":1}
I20260620 07:14:04.809695 23552 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1def444c3b24667aaca2ae35341ea9c" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 34553 } }
I20260620 07:14:04.810226 23552 leader_election.cc:302] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1def444c3b24667aaca2ae35341ea9c; no voters:
I20260620 07:14:04.810848 23552 leader_election.cc:288] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:04.810977 23557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:04.811691 23557 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [term 1 LEADER]: Becoming Leader. State: Replica: c1def444c3b24667aaca2ae35341ea9c, State: Running, Role: LEADER
I20260620 07:14:04.812273 23557 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1def444c3b24667aaca2ae35341ea9c" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 34553 } }
I20260620 07:14:04.812866 23552 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:04.815207 23559 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1def444c3b24667aaca2ae35341ea9c. Latest consensus state: current_term: 1 leader_uuid: "c1def444c3b24667aaca2ae35341ea9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1def444c3b24667aaca2ae35341ea9c" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 34553 } } }
I20260620 07:14:04.815656 23559 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:04.816810 23558 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1def444c3b24667aaca2ae35341ea9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1def444c3b24667aaca2ae35341ea9c" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 34553 } } }
I20260620 07:14:04.817267 23558 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:04.817924 23565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:04.822885 23565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:04.833083 23565 catalog_manager.cc:1347] Generated new cluster ID: 6bf33957d34041e1a03aea1a1dac4835
I20260620 07:14:04.833192 23565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:04.847662 23565 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:04.848985 23565 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:04.856379 23565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c: Generated new TSK 0
I20260620 07:14:04.857190 23565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:14:04.928067 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:34553
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:41921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:05.290597 23578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:05.290863 23578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:05.290956 23578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:05.298130 23578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:05.298231 23578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:05.298409 23578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:14:05.308032 23578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:41921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:34553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:05.309150 23578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:05.310412 23578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:05.323526 23583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:05.323724 23586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:05.323760 23584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:05.324476 23578 server_base.cc:1034] running on GCE node
I20260620 07:14:05.325043 23578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:05.325965 23578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:05.327185 23578 hybrid_clock.cc:648] HybridClock initialized: now 1781939645327137 us; error 58 us; skew 500 ppm
I20260620 07:14:05.327484 23578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:05 dist-test-slave-4n6h krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939645, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:05.330497 23578 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:14:05.331976 23578 webserver.cc:460] Webserver started at http://127.6.53.65:33489/ using document root <none> and password file <none>
I20260620 07:14:05.332643 23578 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:05.332792 23578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:05.333142 23578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:05.336067 23578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "a5eb1b75e63a49e7a21d7493a5212e87"
format_stamp: "Formatted at 2026-06-20 07:14:05 on dist-test-slave-4n6h"
server_key: "fa130bb564c12b715766cf20b72125dc"
server_key_iv: "b73bb26da8662744870107bae645f125"
server_key_version: "encryptionkey@0"
I20260620 07:14:05.336839 23578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "a5eb1b75e63a49e7a21d7493a5212e87"
format_stamp: "Formatted at 2026-06-20 07:14:05 on dist-test-slave-4n6h"
server_key: "fa130bb564c12b715766cf20b72125dc"
server_key_iv: "b73bb26da8662744870107bae645f125"
server_key_version: "encryptionkey@0"
I20260620 07:14:05.341766 23578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 07:14:05.345116 23593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:05.346405 23578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:05.346565 23578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "a5eb1b75e63a49e7a21d7493a5212e87"
format_stamp: "Formatted at 2026-06-20 07:14:05 on dist-test-slave-4n6h"
server_key: "fa130bb564c12b715766cf20b72125dc"
server_key_iv: "b73bb26da8662744870107bae645f125"
server_key_version: "encryptionkey@0"
I20260620 07:14:05.346778 23578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:05.364953 23578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:05.368985 23578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:05.369297 23578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:05.370317 23578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:05.371763 23578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:05.371869 23578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:05.372022 23578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:05.372093 23578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:05.404641 23578 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:38183
I20260620 07:14:05.404681 23705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:38183 every 8 connection(s)
I20260620 07:14:05.405900 23578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:14:05.415334 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 23578
I20260620 07:14:05.415522 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:14:05.416141 6357 external_mini_cluster.cc:1428] Setting key d039219f4eeb015b7d4ce50a9d0b0ff6
Jun 20 07:14:05 dist-test-slave-4n6h krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939645, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:14:05.427453 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:05.408292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:47381 (local address 127.6.53.126:34553)
0620 07:14:05.408760 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:05.408778 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:05.409731 (+ 953us) server_negotiation.cc:407] Connection header received
0620 07:14:05.411886 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:05.411890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:05.411975 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:05.412070 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:05.413659 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:05.414507 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:05.416402 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:05.416919 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:05.417067 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:05.420730 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:05.420739 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:05.420747 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:05.420799 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:05.424358 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:05.425044 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:05.425052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:05.425058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:05.425210 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:05.425594 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:05.425602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:05.425608 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:05.425995 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:05.426148 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:05.426852 (+ 704us) server_negotiation.cc:299] Negotiation successful
0620 07:14:05.427079 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":216,"threads_started":1}
I20260620 07:14:05.428390 23709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:05.408752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:34553 (local address 127.6.53.65:47381)
0620 07:14:05.409569 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:05.409612 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:05.411636 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:05.412326 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:05.412334 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:05.412871 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:05.413490 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:05.413507 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:05.414658 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:05.414671 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:05.416232 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:05.416249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:05.417216 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:05.417222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:05.417529 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:05.418340 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:05.418381 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:05.420517 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:05.424492 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:05.424499 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:05.424511 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:05.424926 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:05.425320 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:05.425323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:05.425326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:05.425493 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:05.426113 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:05.426118 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:05.426480 (+ 362us) client_negotiation.cc:769] Sending connection context
0620 07:14:05.426749 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 07:14:05.427321 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":401,"threads_started":1}
I20260620 07:14:05.429960 23706 heartbeater.cc:347] Connected to a master server at 127.6.53.126:34553
I20260620 07:14:05.430287 23706 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:05.430981 23706 heartbeater.cc:511] Master 127.6.53.126:34553 requested a full tablet report, sending...
I20260620 07:14:05.433079 23517 ts_manager.cc:194] Registered new tserver with Master: a5eb1b75e63a49e7a21d7493a5212e87 (127.6.53.65:38183)
I20260620 07:14:05.434646 23517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:47381
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:14:05.469415 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:34553
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:41921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:05.788617 23712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:05.788862 23712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:05.789005 23712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:05.796227 23712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:05.796329 23712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:05.796533 23712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:14:05.806188 23712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:41921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:34553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:05.807458 23712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:05.808692 23712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:05.821761 23717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:05.821863 23718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:05.821883 23720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:05.822813 23712 server_base.cc:1034] running on GCE node
I20260620 07:14:05.823333 23712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:05.824184 23712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:05.825412 23712 hybrid_clock.cc:648] HybridClock initialized: now 1781939645825359 us; error 40 us; skew 500 ppm
I20260620 07:14:05.825735 23712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:05 dist-test-slave-4n6h krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939645, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:05.828763 23712 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:14:05.830230 23712 webserver.cc:460] Webserver started at http://127.6.53.66:36235/ using document root <none> and password file <none>
I20260620 07:14:05.830929 23712 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:05.831053 23712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:05.831357 23712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:05.834163 23712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "78d879f764894965b3be2e7eae7deba8"
format_stamp: "Formatted at 2026-06-20 07:14:05 on dist-test-slave-4n6h"
server_key: "be3b671fde13cedc8d71e0e65ce26171"
server_key_iv: "28c2d64aa638d15854f778b92a8dffdd"
server_key_version: "encryptionkey@0"
I20260620 07:14:05.835369 23712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "78d879f764894965b3be2e7eae7deba8"
format_stamp: "Formatted at 2026-06-20 07:14:05 on dist-test-slave-4n6h"
server_key: "be3b671fde13cedc8d71e0e65ce26171"
server_key_iv: "28c2d64aa638d15854f778b92a8dffdd"
server_key_version: "encryptionkey@0"
I20260620 07:14:05.841928 23712 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 07:14:05.845191 23727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:05.846536 23712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:05.846704 23712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "78d879f764894965b3be2e7eae7deba8"
format_stamp: "Formatted at 2026-06-20 07:14:05 on dist-test-slave-4n6h"
server_key: "be3b671fde13cedc8d71e0e65ce26171"
server_key_iv: "28c2d64aa638d15854f778b92a8dffdd"
server_key_version: "encryptionkey@0"
I20260620 07:14:05.846915 23712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:05.870332 23712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:05.874370 23712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:05.874648 23712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:05.875578 23712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:05.876842 23712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:05.876945 23712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:05.877100 23712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:05.877173 23712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:05.910019 23712 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:35443
I20260620 07:14:05.910108 23839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:35443 every 8 connection(s)
I20260620 07:14:05.911612 23712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:14:05.921703 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 23712
I20260620 07:14:05.921907 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:14:05.922489 6357 external_mini_cluster.cc:1428] Setting key 94114d35f439e4f6a75bcacc76c84b5b
Jun 20 07:14:05 dist-test-slave-4n6h krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939645, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:14:05.933722 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:05.914305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:38771 (local address 127.6.53.126:34553)
0620 07:14:05.914507 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:05.914513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:05.915684 (+ 1171us) server_negotiation.cc:407] Connection header received
0620 07:14:05.917523 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:05.917528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:05.917594 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:05.917694 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:05.919345 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:05.920159 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:05.921983 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:05.922481 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:05.922626 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:05.926480 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:05.926491 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:05.926499 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:05.926550 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:05.930200 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:05.930955 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:05.930964 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:05.930971 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:05.931164 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:05.931576 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:05.931584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:05.931590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:05.931979 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:05.932156 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:05.932878 (+ 722us) server_negotiation.cc:299] Negotiation successful
0620 07:14:05.933168 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:14:05.934485 23842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:05.914757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:34553 (local address 127.6.53.66:38771)
0620 07:14:05.915522 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:05.915571 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:05.917269 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:05.917881 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:05.917889 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:05.918539 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:05.919184 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:05.919203 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:05.920302 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:05.920306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:05.921803 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:05.921821 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:05.922725 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:05.922733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:05.923025 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:05.923820 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:05.923843 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:05.926195 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:05.930374 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:05.930380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:05.930392 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:05.930810 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:05.931282 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:05.931285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:05.931287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:05.931450 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:05.932112 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:05.932117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:05.932421 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 07:14:05.932731 (+ 310us) client_negotiation.cc:240] Negotiation successful
0620 07:14:05.933100 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":323,"threads_started":1}
I20260620 07:14:05.935933 23840 heartbeater.cc:347] Connected to a master server at 127.6.53.126:34553
I20260620 07:14:05.936218 23840 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:05.936869 23840 heartbeater.cc:511] Master 127.6.53.126:34553 requested a full tablet report, sending...
I20260620 07:14:05.938221 23517 ts_manager.cc:194] Registered new tserver with Master: 78d879f764894965b3be2e7eae7deba8 (127.6.53.66:35443)
I20260620 07:14:05.939052 23517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:38771
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:14:05.975443 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:34553
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:41921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:06.294109 23845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:06.294441 23845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:06.294605 23845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:06.301779 23845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:06.301913 23845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:06.302070 23845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:14:06.311327 23845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:41921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:34553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:06.312456 23845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:06.313685 23845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:06.326001 23850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:06.326037 23851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:06.326072 23853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:06.326663 23845 server_base.cc:1034] running on GCE node
I20260620 07:14:06.327235 23845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:06.328042 23845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:06.329269 23845 hybrid_clock.cc:648] HybridClock initialized: now 1781939646329229 us; error 47 us; skew 500 ppm
I20260620 07:14:06.329562 23845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:06 dist-test-slave-4n6h krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939646, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:06.333245 23845 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:14:06.334856 23845 webserver.cc:460] Webserver started at http://127.6.53.67:34841/ using document root <none> and password file <none>
I20260620 07:14:06.335546 23845 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:06.335685 23845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:06.336006 23845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:06.338894 23845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "afd690f5a68b48a3a9622d4d692cd218"
format_stamp: "Formatted at 2026-06-20 07:14:06 on dist-test-slave-4n6h"
server_key: "3237ea1b3ac064418f6c2e21936eb880"
server_key_iv: "edba69c667db52dc54641ffeab2a462c"
server_key_version: "encryptionkey@0"
I20260620 07:14:06.339702 23845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "afd690f5a68b48a3a9622d4d692cd218"
format_stamp: "Formatted at 2026-06-20 07:14:06 on dist-test-slave-4n6h"
server_key: "3237ea1b3ac064418f6c2e21936eb880"
server_key_iv: "edba69c667db52dc54641ffeab2a462c"
server_key_version: "encryptionkey@0"
I20260620 07:14:06.344666 23845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:14:06.348028 23860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:06.349272 23845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:14:06.349459 23845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "afd690f5a68b48a3a9622d4d692cd218"
format_stamp: "Formatted at 2026-06-20 07:14:06 on dist-test-slave-4n6h"
server_key: "3237ea1b3ac064418f6c2e21936eb880"
server_key_iv: "edba69c667db52dc54641ffeab2a462c"
server_key_version: "encryptionkey@0"
I20260620 07:14:06.349653 23845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:06.377254 23845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:06.381490 23845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:06.381913 23845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:06.382942 23845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:06.384339 23845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:06.384421 23845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:06.384616 23845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:06.384675 23845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:06.420171 23845 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:44085
I20260620 07:14:06.420261 23972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:44085 every 8 connection(s)
I20260620 07:14:06.421425 23845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:14:06.426966 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 23845
I20260620 07:14:06.427150 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:14:06.427577 6357 external_mini_cluster.cc:1428] Setting key 181dc03110ea4e6ba546040bb94492aa
Jun 20 07:14:06 dist-test-slave-4n6h krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939646, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:14:06.437422 23706 heartbeater.cc:503] Master 127.6.53.126:34553 was elected leader, sending a full tablet report...
I20260620 07:14:06.441831 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.424284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:38405 (local address 127.6.53.126:34553)
0620 07:14:06.424470 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:06.424475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:06.425473 (+ 998us) server_negotiation.cc:407] Connection header received
0620 07:14:06.427444 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:06.427450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:06.427570 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:06.427683 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:06.429331 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.430136 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.431746 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.432141 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.432232 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.435962 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:06.435966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:06.435969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:06.436000 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:06.438866 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:06.439642 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:06.439646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:06.439649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:06.439769 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:06.440126 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:06.440129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:06.440130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:06.440383 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:06.440487 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:06.441321 (+ 834us) server_negotiation.cc:299] Negotiation successful
0620 07:14:06.441530 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:14:06.443086 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.424601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:34553 (local address 127.6.53.67:38405)
0620 07:14:06.425307 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:06.425353 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:06.427159 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:06.427903 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:06.427911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:06.428530 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:06.429170 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:06.429188 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.430324 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.430329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:06.431608 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:06.431621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.432395 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.432402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:06.432631 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.433389 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:06.433431 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:06.435746 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:06.439017 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:06.439043 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:06.439060 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:06.439533 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:06.439859 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:06.439863 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:06.439865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:06.440032 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:06.440482 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:06.440488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:06.440871 (+ 383us) client_negotiation.cc:769] Sending connection context
0620 07:14:06.441183 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 07:14:06.441679 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":305,"threads_started":1}
I20260620 07:14:06.444340 23973 heartbeater.cc:347] Connected to a master server at 127.6.53.126:34553
I20260620 07:14:06.444626 23973 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:06.445312 23973 heartbeater.cc:511] Master 127.6.53.126:34553 requested a full tablet report, sending...
I20260620 07:14:06.446760 23517 ts_manager.cc:194] Registered new tserver with Master: afd690f5a68b48a3a9622d4d692cd218 (127.6.53.67:44085)
I20260620 07:14:06.447566 23517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:38405
I20260620 07:14:06.454952 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:14:06.471262 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.459275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53540 (local address 127.6.53.126:34553)
0620 07:14:06.459430 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:06.459435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:06.459871 (+ 436us) server_negotiation.cc:407] Connection header received
0620 07:14:06.460000 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:06.460003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:06.460060 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:06.460138 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:06.461501 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.462206 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.463582 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.464073 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.464167 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.466599 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:06.466604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:06.466607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:06.466636 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:06.469122 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:06.469661 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:06.469665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:06.469667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:06.469797 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:06.470084 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:06.470087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:06.470089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:06.470450 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:06.470625 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:06.470895 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 07:14:06.471039 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:14:06.474951 23517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:14:06.481554 23517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:14:06.505688 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.493086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:44085 (local address 127.0.0.1:48978)
0620 07:14:06.495684 (+ 2598us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:06.495701 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:06.495819 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:06.496851 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:06.496859 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:06.496894 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:06.497397 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:06.497415 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.499604 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.499609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:06.502284 (+ 2675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:06.502301 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.504576 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.504583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:06.504788 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.504875 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 07:14:06.505014 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 07:14:06.505215 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2467,"mutex_wait_us":50,"thread_start_us":2333,"threads_started":1}
I20260620 07:14:06.506721 23988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.493869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48978 (local address 127.6.53.67:44085)
0620 07:14:06.495267 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:06.495275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:06.495941 (+ 666us) server_negotiation.cc:407] Connection header received
0620 07:14:06.496041 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:06.496050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:06.496243 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:06.496419 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:06.497641 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.499423 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.502480 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.504015 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.504162 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.504441 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:06.506110 (+ 1669us) server_negotiation.cc:299] Negotiation successful
0620 07:14:06.506341 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1222,"thread_start_us":1040,"threads_started":1}
I20260620 07:14:06.507570 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.492987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:35443 (local address 127.0.0.1:52604)
0620 07:14:06.494576 (+ 1589us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:06.494621 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:06.494789 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:06.497441 (+ 2652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:06.497454 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:06.497482 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:06.497982 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:06.497996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.501050 (+ 3054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.501058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:06.503344 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:06.503359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.506685 (+ 3326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.506695 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:06.506901 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.506934 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:14:06.507026 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 07:14:06.507139 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"mutex_wait_us":4,"thread_start_us":1047,"threads_started":1}
I20260620 07:14:06.508239 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.493439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.6.53.66:35443)
0620 07:14:06.495376 (+ 1937us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:06.495385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:06.495414 (+ 29us) server_negotiation.cc:407] Connection header received
0620 07:14:06.495515 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:06.495532 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:06.495785 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:06.497210 (+ 1425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:06.498246 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.500875 (+ 2629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.503575 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.506155 (+ 2580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.506321 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.506516 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:06.507715 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0620 07:14:06.507890 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1774,"thread_start_us":1722,"threads_started":1}
I20260620 07:14:06.512305 23775 tablet_service.cc:1467] Processing CreateTablet for tablet dd8249eabaaa48789dee1cedc1357f66 (DEFAULT_TABLE table=test-table [id=bace73a559404f9d8f03f3ab0987b151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:06.512305 23908 tablet_service.cc:1467] Processing CreateTablet for tablet dd8249eabaaa48789dee1cedc1357f66 (DEFAULT_TABLE table=test-table [id=bace73a559404f9d8f03f3ab0987b151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:06.513752 23775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8249eabaaa48789dee1cedc1357f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:06.513752 23908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8249eabaaa48789dee1cedc1357f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:06.516018 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.495781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38183 (local address 127.0.0.1:49734)
0620 07:14:06.507987 (+ 12206us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:06.508006 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:06.508118 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:06.510278 (+ 2160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:06.510284 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:06.510311 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:06.510485 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:06.510499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.511889 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.511896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:06.513257 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:06.513270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.515290 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.515296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:06.515507 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.515558 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 07:14:06.515654 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 07:14:06.515792 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12110,"thread_start_us":12792,"threads_started":1}
I20260620 07:14:06.516685 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.496024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49734 (local address 127.6.53.65:38183)
0620 07:14:06.509752 (+ 13728us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:06.509765 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:06.509795 (+ 30us) server_negotiation.cc:407] Connection header received
0620 07:14:06.509894 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:06.509899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:06.510069 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:06.510292 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:06.510669 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.511739 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.513437 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.514818 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.514957 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.515196 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:06.516283 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0620 07:14:06.516441 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12241,"thread_start_us":12387,"threads_started":1}
I20260620 07:14:06.521752 23641 tablet_service.cc:1467] Processing CreateTablet for tablet dd8249eabaaa48789dee1cedc1357f66 (DEFAULT_TABLE table=test-table [id=bace73a559404f9d8f03f3ab0987b151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:06.523243 23641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8249eabaaa48789dee1cedc1357f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:06.528489 23991 tablet_bootstrap.cc:492] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218: Bootstrap starting.
I20260620 07:14:06.529325 23992 tablet_bootstrap.cc:492] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8: Bootstrap starting.
I20260620 07:14:06.532891 23991 tablet_bootstrap.cc:654] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:06.533111 23992 tablet_bootstrap.cc:654] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:06.534094 23991 log.cc:824] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:06.534379 23992 log.cc:824] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:06.536242 23995 tablet_bootstrap.cc:492] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87: Bootstrap starting.
I20260620 07:14:06.537175 23991 tablet_bootstrap.cc:492] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218: No bootstrap required, opened a new log
I20260620 07:14:06.537305 23992 tablet_bootstrap.cc:492] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8: No bootstrap required, opened a new log
I20260620 07:14:06.537492 23991 ts_tablet_manager.cc:1397] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 07:14:06.537575 23992 ts_tablet_manager.cc:1397] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 07:14:06.539666 23995 tablet_bootstrap.cc:654] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:06.540809 23995 log.cc:824] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:06.544840 23995 tablet_bootstrap.cc:492] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87: No bootstrap required, opened a new log
I20260620 07:14:06.545130 23995 ts_tablet_manager.cc:1397] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 07:14:06.546171 23992 raft_consensus.cc:348] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d879f764894965b3be2e7eae7deba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 35443 } } peers { permanent_uuid: "afd690f5a68b48a3a9622d4d692cd218" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 44085 } } peers { permanent_uuid: "a5eb1b75e63a49e7a21d7493a5212e87" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38183 } }
I20260620 07:14:06.546605 23992 raft_consensus.cc:374] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:06.546728 23992 raft_consensus.cc:729] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d879f764894965b3be2e7eae7deba8, State: Initialized, Role: FOLLOWER
I20260620 07:14:06.546762 23991 raft_consensus.cc:348] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d879f764894965b3be2e7eae7deba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 35443 } } peers { permanent_uuid: "afd690f5a68b48a3a9622d4d692cd218" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 44085 } } peers { permanent_uuid: "a5eb1b75e63a49e7a21d7493a5212e87" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38183 } }
I20260620 07:14:06.547219 23991 raft_consensus.cc:374] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:06.547320 23991 raft_consensus.cc:729] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd690f5a68b48a3a9622d4d692cd218, State: Initialized, Role: FOLLOWER
I20260620 07:14:06.547382 23992 consensus_queue.cc:260] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d879f764894965b3be2e7eae7deba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 35443 } } peers { permanent_uuid: "afd690f5a68b48a3a9622d4d692cd218" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 44085 } } peers { permanent_uuid: "a5eb1b75e63a49e7a21d7493a5212e87" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38183 } }
I20260620 07:14:06.547899 23991 consensus_queue.cc:260] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d879f764894965b3be2e7eae7deba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 35443 } } peers { permanent_uuid: "afd690f5a68b48a3a9622d4d692cd218" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 44085 } } peers { permanent_uuid: "a5eb1b75e63a49e7a21d7493a5212e87" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38183 } }
I20260620 07:14:06.548578 23840 heartbeater.cc:503] Master 127.6.53.126:34553 was elected leader, sending a full tablet report...
I20260620 07:14:06.549226 23992 ts_tablet_manager.cc:1428] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 07:14:06.550295 23973 heartbeater.cc:503] Master 127.6.53.126:34553 was elected leader, sending a full tablet report...
I20260620 07:14:06.551052 23991 ts_tablet_manager.cc:1428] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 07:14:06.556404 23995 raft_consensus.cc:348] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d879f764894965b3be2e7eae7deba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 35443 } } peers { permanent_uuid: "afd690f5a68b48a3a9622d4d692cd218" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 44085 } } peers { permanent_uuid: "a5eb1b75e63a49e7a21d7493a5212e87" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38183 } }
I20260620 07:14:06.556766 23995 raft_consensus.cc:374] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:06.556886 23995 raft_consensus.cc:729] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5eb1b75e63a49e7a21d7493a5212e87, State: Initialized, Role: FOLLOWER
I20260620 07:14:06.557473 23995 consensus_queue.cc:260] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d879f764894965b3be2e7eae7deba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 35443 } } peers { permanent_uuid: "afd690f5a68b48a3a9622d4d692cd218" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 44085 } } peers { permanent_uuid: "a5eb1b75e63a49e7a21d7493a5212e87" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38183 } }
I20260620 07:14:06.558874 23995 ts_tablet_manager.cc:1428] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
W20260620 07:14:06.658330 23707 tablet.cc:2369] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:14:06.663745 23841 tablet.cc:2369] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:14:06.673069 23974 tablet.cc:2369] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:14:06.804459 23997 raft_consensus.cc:482] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:06.804821 23997 raft_consensus.cc:504] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d879f764894965b3be2e7eae7deba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 35443 } } peers { permanent_uuid: "afd690f5a68b48a3a9622d4d692cd218" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 44085 } } peers { permanent_uuid: "a5eb1b75e63a49e7a21d7493a5212e87" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38183 } }
I20260620 07:14:06.806600 23997 leader_election.cc:288] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afd690f5a68b48a3a9622d4d692cd218 (127.6.53.67:44085), a5eb1b75e63a49e7a21d7493a5212e87 (127.6.53.65:38183)
I20260620 07:14:06.814630 23999 raft_consensus.cc:482] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:06.815361 23988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.807014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:45775 (local address 127.6.53.67:44085)
0620 07:14:06.807230 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:06.807237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:06.807788 (+ 551us) server_negotiation.cc:407] Connection header received
0620 07:14:06.808073 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:06.808082 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:06.808179 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:06.808290 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:06.808947 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.809959 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.812858 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.814083 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.814198 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.814330 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:06.814980 (+ 650us) server_negotiation.cc:299] Negotiation successful
0620 07:14:06.815115 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:14:06.815172 23999 raft_consensus.cc:504] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d879f764894965b3be2e7eae7deba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 35443 } } peers { permanent_uuid: "afd690f5a68b48a3a9622d4d692cd218" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 44085 } } peers { permanent_uuid: "a5eb1b75e63a49e7a21d7493a5212e87" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38183 } }
I20260620 07:14:06.815821 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.807845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38183 (local address 127.6.53.66:38293)
0620 07:14:06.808412 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:06.808428 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:06.808555 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:06.808853 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:06.808859 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:06.808887 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:06.809282 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:06.809296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.810840 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.810846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:06.812327 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:06.812339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.814505 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.814510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:06.814776 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.814843 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 07:14:06.815205 (+ 362us) client_negotiation.cc:240] Negotiation successful
0620 07:14:06.815328 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":79,"thread_start_us":470,"threads_started":1}
I20260620 07:14:06.816066 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.807031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:44085 (local address 127.6.53.66:45775)
0620 07:14:06.807667 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:06.807683 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:06.807884 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:06.808317 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:06.808324 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:06.808350 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:06.808767 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:06.808800 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.810153 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.810158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:06.812681 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:06.812698 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.814499 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.814505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:06.814768 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.814821 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 07:14:06.815538 (+ 717us) client_negotiation.cc:240] Negotiation successful
0620 07:14:06.815668 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":27,"thread_start_us":354,"threads_started":1}
I20260620 07:14:06.817108 23928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8249eabaaa48789dee1cedc1357f66" candidate_uuid: "78d879f764894965b3be2e7eae7deba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afd690f5a68b48a3a9622d4d692cd218" is_pre_election: true
I20260620 07:14:06.817672 23928 raft_consensus.cc:2454] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78d879f764894965b3be2e7eae7deba8 in term 0.
I20260620 07:14:06.818189 23999 leader_election.cc:288] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78d879f764894965b3be2e7eae7deba8 (127.6.53.66:35443), afd690f5a68b48a3a9622d4d692cd218 (127.6.53.67:44085)
I20260620 07:14:06.818648 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.808120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:38293 (local address 127.6.53.65:38183)
0620 07:14:06.808318 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:06.808324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:06.808529 (+ 205us) server_negotiation.cc:407] Connection header received
0620 07:14:06.808673 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:06.808677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:06.808733 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:06.808804 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:06.809473 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.810648 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.812510 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.814046 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.814198 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.814364 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:06.814980 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 07:14:06.815131 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 07:14:06.819096 23729 leader_election.cc:302] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78d879f764894965b3be2e7eae7deba8, afd690f5a68b48a3a9622d4d692cd218; no voters:
I20260620 07:14:06.819519 23997 raft_consensus.cc:2781] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:14:06.819676 23997 raft_consensus.cc:482] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:14:06.819808 23997 raft_consensus.cc:3037] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:06.819561 23661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8249eabaaa48789dee1cedc1357f66" candidate_uuid: "78d879f764894965b3be2e7eae7deba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5eb1b75e63a49e7a21d7493a5212e87" is_pre_election: true
I20260620 07:14:06.820101 23661 raft_consensus.cc:2454] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78d879f764894965b3be2e7eae7deba8 in term 0.
I20260620 07:14:06.823541 23997 raft_consensus.cc:504] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d879f764894965b3be2e7eae7deba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 35443 } } peers { permanent_uuid: "afd690f5a68b48a3a9622d4d692cd218" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 44085 } } peers { permanent_uuid: "a5eb1b75e63a49e7a21d7493a5212e87" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38183 } }
I20260620 07:14:06.824239 23997 leader_election.cc:288] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [CANDIDATE]: Term 1 election: Requested vote from peers afd690f5a68b48a3a9622d4d692cd218 (127.6.53.67:44085), a5eb1b75e63a49e7a21d7493a5212e87 (127.6.53.65:38183)
I20260620 07:14:06.824771 23661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8249eabaaa48789dee1cedc1357f66" candidate_uuid: "78d879f764894965b3be2e7eae7deba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5eb1b75e63a49e7a21d7493a5212e87"
I20260620 07:14:06.824959 23661 raft_consensus.cc:3037] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:06.826107 23928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8249eabaaa48789dee1cedc1357f66" candidate_uuid: "78d879f764894965b3be2e7eae7deba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afd690f5a68b48a3a9622d4d692cd218"
I20260620 07:14:06.826331 23928 raft_consensus.cc:3037] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:06.827358 23988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.819193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:48843 (local address 127.6.53.67:44085)
0620 07:14:06.820234 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:06.820240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:06.820319 (+ 79us) server_negotiation.cc:407] Connection header received
0620 07:14:06.820590 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:06.820595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:06.820673 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:06.820769 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:06.821255 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.821964 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.824501 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.825734 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.825829 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.825928 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:06.826930 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0620 07:14:06.827069 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917}
I20260620 07:14:06.828058 24003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.818708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:44085 (local address 127.6.53.65:48843)
0620 07:14:06.820216 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:06.820242 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:06.820429 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:06.820783 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:06.820788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:06.820818 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:06.821128 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:06.821136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.822088 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.822095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:06.824330 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:06.824345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.826516 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.826525 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:06.826702 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.826745 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 07:14:06.827569 (+ 824us) client_negotiation.cc:240] Negotiation successful
0620 07:14:06.827713 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1360,"thread_start_us":3713,"threads_started":1}
I20260620 07:14:06.828341 23661 raft_consensus.cc:2454] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78d879f764894965b3be2e7eae7deba8 in term 1.
I20260620 07:14:06.828840 23927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8249eabaaa48789dee1cedc1357f66" candidate_uuid: "a5eb1b75e63a49e7a21d7493a5212e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afd690f5a68b48a3a9622d4d692cd218" is_pre_election: true
I20260620 07:14:06.829133 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.818743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:57549 (local address 127.6.53.66:35443)
0620 07:14:06.819268 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:06.819274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:06.822875 (+ 3601us) server_negotiation.cc:407] Connection header received
0620 07:14:06.823007 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:06.823012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:06.823077 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:06.823166 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:06.823817 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.824839 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.826780 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.827989 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.828112 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.828211 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:06.828781 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 07:14:06.828901 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394}
I20260620 07:14:06.829174 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.819513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:35443 (local address 127.6.53.65:57549)
0620 07:14:06.822764 (+ 3251us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:06.822786 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:06.822907 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:06.823200 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:06.823206 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:06.823237 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:06.823619 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:06.823629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.825051 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.825056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:06.826613 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:06.826626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.828429 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.828436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:06.828605 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.828638 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:14:06.828750 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 07:14:06.828873 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3100,"thread_start_us":358,"threads_started":1}
I20260620 07:14:06.829576 23928 raft_consensus.cc:2454] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78d879f764894965b3be2e7eae7deba8 in term 1.
I20260620 07:14:06.829613 23729 leader_election.cc:302] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78d879f764894965b3be2e7eae7deba8, a5eb1b75e63a49e7a21d7493a5212e87; no voters:
I20260620 07:14:06.830014 23997 raft_consensus.cc:2781] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:06.830636 23795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8249eabaaa48789dee1cedc1357f66" candidate_uuid: "a5eb1b75e63a49e7a21d7493a5212e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78d879f764894965b3be2e7eae7deba8" is_pre_election: true
I20260620 07:14:06.831493 23997 raft_consensus.cc:686] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [term 1 LEADER]: Becoming Leader. State: Replica: 78d879f764894965b3be2e7eae7deba8, State: Running, Role: LEADER
I20260620 07:14:06.831954 23997 consensus_queue.cc:237] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d879f764894965b3be2e7eae7deba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 35443 } } peers { permanent_uuid: "afd690f5a68b48a3a9622d4d692cd218" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 44085 } } peers { permanent_uuid: "a5eb1b75e63a49e7a21d7493a5212e87" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38183 } }
I20260620 07:14:06.835563 23597 leader_election.cc:302] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5eb1b75e63a49e7a21d7493a5212e87; no voters: 78d879f764894965b3be2e7eae7deba8, afd690f5a68b48a3a9622d4d692cd218
I20260620 07:14:06.835940 23999 raft_consensus.cc:2726] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 07:14:06.836968 23517 catalog_manager.cc:5495] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78d879f764894965b3be2e7eae7deba8 (127.6.53.66). New cstate: current_term: 1 leader_uuid: "78d879f764894965b3be2e7eae7deba8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d879f764894965b3be2e7eae7deba8" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 35443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afd690f5a68b48a3a9622d4d692cd218" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 44085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5eb1b75e63a49e7a21d7493a5212e87" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38183 } health_report { overall_health: UNKNOWN } } }
I20260620 07:14:06.868148 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.863211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.6.53.66:35443)
0620 07:14:06.863388 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:06.863392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:06.863409 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:14:06.863594 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:06.863598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:06.863689 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:06.863820 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:14:06.864149 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.864882 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.866212 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.866663 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.866764 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.867193 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:14:06.867667 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:14:06.867786 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:06.867869 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 07:14:06.867932 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:14:06.875701 23928 raft_consensus.cc:1261] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218 [term 1 FOLLOWER]: Refusing update from remote peer 78d879f764894965b3be2e7eae7deba8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:06.875819 23661 raft_consensus.cc:1261] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87 [term 1 FOLLOWER]: Refusing update from remote peer 78d879f764894965b3be2e7eae7deba8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:06.876585 24004 consensus_queue.cc:1035] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afd690f5a68b48a3a9622d4d692cd218" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:06.877012 23997 consensus_queue.cc:1035] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5eb1b75e63a49e7a21d7493a5212e87" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:06.902961 23515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:06.903329 23515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:06.906821 23515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1def444c3b24667aaca2ae35341ea9c: Sending DeleteTablet for 3 replicas of tablet dd8249eabaaa48789dee1cedc1357f66
I20260620 07:14:06.907891 23775 tablet_service.cc:1514] Processing DeleteTablet for tablet dd8249eabaaa48789dee1cedc1357f66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:06 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:52604
I20260620 07:14:06.908006 23908 tablet_service.cc:1514] Processing DeleteTablet for tablet dd8249eabaaa48789dee1cedc1357f66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:06 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:48978
I20260620 07:14:06.908088 23641 tablet_service.cc:1514] Processing DeleteTablet for tablet dd8249eabaaa48789dee1cedc1357f66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:06 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:49734
I20260620 07:14:06.909376 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 23578
I20260620 07:14:06.929740 24018 tablet_replica.cc:330] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218: stopping tablet replica
I20260620 07:14:06.930176 24018 raft_consensus.cc:2229] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:06.931034 24018 raft_consensus.cc:2258] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:06.933954 24018 ts_tablet_manager.cc:1905] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:06.941218 24018 ts_tablet_manager.cc:1918] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:06.941463 24018 log.cc:1196] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/dd8249eabaaa48789dee1cedc1357f66
I20260620 07:14:06.942145 24018 ts_tablet_manager.cc:1939] T dd8249eabaaa48789dee1cedc1357f66 P afd690f5a68b48a3a9622d4d692cd218: Deleting consensus metadata
I20260620 07:14:06.944296 23502 catalog_manager.cc:4841] TS afd690f5a68b48a3a9622d4d692cd218 (127.6.53.67:44085): tablet dd8249eabaaa48789dee1cedc1357f66 (table test-table [id=bace73a559404f9d8f03f3ab0987b151]) successfully deleted
I20260620 07:14:06.950811 24019 tablet_replica.cc:330] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87: stopping tablet replica
I20260620 07:14:06.951393 24019 raft_consensus.cc:2229] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:06.952024 24019 raft_consensus.cc:2258] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:06.952764 24017 tablet_replica.cc:330] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8: stopping tablet replica
I20260620 07:14:06.953293 24017 raft_consensus.cc:2229] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [term 1 LEADER]: Raft consensus shutting down.
Jun 20 07:14:06 dist-test-slave-4n6h krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:14:06.954020 24017 raft_consensus.cc:2258] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:06.956287 24017 ts_tablet_manager.cc:1905] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:06.956393 24019 ts_tablet_manager.cc:1905] T dd8249eabaaa48789dee1cedc1357f66 P a5eb1b75e63a49e7a21d7493a5212e87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:06.963877 24017 ts_tablet_manager.cc:1918] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:06.964124 24017 log.cc:1196] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/dd8249eabaaa48789dee1cedc1357f66
I20260620 07:14:06.964757 24017 ts_tablet_manager.cc:1939] T dd8249eabaaa48789dee1cedc1357f66 P 78d879f764894965b3be2e7eae7deba8: Deleting consensus metadata
I20260620 07:14:06.965867 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:06.911894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49742 (local address 127.6.53.65:38183)
0620 07:14:06.912122 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:06.912132 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:06.933207 (+ 21075us) server_negotiation.cc:407] Connection header received
0620 07:14:06.934416 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:06.934426 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:06.934573 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:06.934677 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:06.939326 (+ 4649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.940896 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.946340 (+ 5444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:06.947300 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:06.947430 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:06.955383 (+ 7953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:06.955397 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:06.955413 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:06.955457 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:06.959194 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:06.962381 (+ 3187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:06.962390 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:06.962393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:06.962560 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:06.963026 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:06.963031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:06.963035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:06.963463 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:06.963763 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:06.964402 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 07:14:06.964655 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:14:06.966578 23504 catalog_manager.cc:4841] TS 78d879f764894965b3be2e7eae7deba8 (127.6.53.66:35443): tablet dd8249eabaaa48789dee1cedc1357f66 (table test-table [id=bace73a559404f9d8f03f3ab0987b151]) successfully deleted
I20260620 07:14:06.966763 23701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:07.059265 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 23578
I20260620 07:14:07.077759 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 23712
W20260620 07:14:07.077917 23502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.53.65:38183 (error 108)
I20260620 07:14:07.080283 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:07.079476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38183 (local address 127.0.0.1:49750)
0620 07:14:07.079922 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:07.080060 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:38183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":224,"threads_started":1}
W20260620 07:14:07.080580 23502 catalog_manager.cc:4584] TS a5eb1b75e63a49e7a21d7493a5212e87 (127.6.53.65:38183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dd8249eabaaa48789dee1cedc1357f66: Network error: Client connection negotiation failed: client connection to 127.6.53.65:38183: connect: Connection refused (error 111)
Jun 20 07:14:07 dist-test-slave-4n6h krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:14:07.096390 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:07.078552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.6.53.66:35443)
0620 07:14:07.078750 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:07.078759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:07.078924 (+ 165us) server_negotiation.cc:407] Connection header received
0620 07:14:07.079226 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:07.079232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:07.079335 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:07.079408 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:07.081253 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:07.082395 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:07.083693 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:07.084489 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:07.084643 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:07.088770 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:07.088793 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:07.088814 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:07.088865 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:07.093142 (+ 4277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:07.093920 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:07.093931 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:07.093935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:07.094144 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:07.094600 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:07.094607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:07.094611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:07.095171 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:07.095508 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:07.095683 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 07:14:07.095945 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:14:07.097237 23835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:07.134341 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:07.133783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38183 (local address 127.0.0.1:49764)
0620 07:14:07.134007 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:07.134134 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:38183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 07:14:07.135494 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:07.135034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38183 (local address 127.0.0.1:49776)
0620 07:14:07.135245 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:07.135336 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:38183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 07:14:07.196244 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:07.195639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38183 (local address 127.0.0.1:49782)
0620 07:14:07.195897 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:07.196010 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:38183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 07:14:07.197407 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:07.196976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38183 (local address 127.0.0.1:49792)
0620 07:14:07.197155 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:07.197239 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:38183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 07:14:07.208526 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 23712
I20260620 07:14:07.227315 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 23845
Jun 20 07:14:07 dist-test-slave-4n6h krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:14:07.241210 23988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:07.228001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.6.53.67:44085)
0620 07:14:07.228167 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:07.228171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:07.228222 (+ 51us) server_negotiation.cc:407] Connection header received
0620 07:14:07.228358 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:07.228362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:07.228422 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:07.228508 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:07.229694 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:07.230427 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:07.231632 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:07.232040 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:07.232129 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:07.236500 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:07.236513 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:07.236529 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:07.236561 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:07.239090 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:07.239641 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:07.239648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:07.239650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:07.239777 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:07.240067 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:07.240070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:07.240072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:07.240386 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:07.240585 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:07.240846 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 07:14:07.240990 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:14:07.241838 23968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:07.277995 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:07.277489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38183 (local address 127.0.0.1:49806)
0620 07:14:07.277774 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:07.277858 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:38183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260620 07:14:07.279049 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:07.278652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38183 (local address 127.0.0.1:49820)
0620 07:14:07.278852 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:07.278925 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:38183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 07:14:07.329427 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 23845
I20260620 07:14:07.345515 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 23485
I20260620 07:14:07.346750 23547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:07.453634 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:07.453034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38183 (local address 127.0.0.1:49832)
0620 07:14:07.453279 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:07.453429 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:38183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 07:14:07.453787 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 23485
2026-06-20T07:14:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4507 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T07:14:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:07Z Disabled control of system clock
I20260620 07:14:07.527086 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:45199
--webserver_interface=127.6.53.126
--webserver_port=0
--builtin_ntp_servers=127.6.53.84:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:45199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:14:07.849983 24030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:07.850440 24030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:07.850546 24030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:07.858006 24030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:07.858126 24030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:07.858170 24030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:07.858227 24030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:07.858346 24030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:07.868207 24030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:45199
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:45199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:07.869413 24030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:07.871174 24030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:07.882961 24035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:07.883052 24036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:07.883189 24038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:07.884220 24030 server_base.cc:1034] running on GCE node
I20260620 07:14:07.885004 24030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:07.886423 24030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:07.887683 24030 hybrid_clock.cc:648] HybridClock initialized: now 1781939647887641 us; error 53 us; skew 500 ppm
I20260620 07:14:07.888021 24030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:14:07.893432 24030 webserver.cc:460] Webserver started at http://127.6.53.126:45297/ using document root <none> and password file <none>
I20260620 07:14:07.894129 24030 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:07.894261 24030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:07.894624 24030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:07.897459 24030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "3c59cdc5715a45768fc27cfbaaab2fd7"
format_stamp: "Formatted at 2026-06-20 07:14:07 on dist-test-slave-4n6h"
server_key: "9db4269af69e2936ce97c13b41e1b92f"
server_key_iv: "099f3b50a885f9611343887df3b94c9b"
server_key_version: "encryptionkey@0"
I20260620 07:14:07.898240 24030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "3c59cdc5715a45768fc27cfbaaab2fd7"
format_stamp: "Formatted at 2026-06-20 07:14:07 on dist-test-slave-4n6h"
server_key: "9db4269af69e2936ce97c13b41e1b92f"
server_key_iv: "099f3b50a885f9611343887df3b94c9b"
server_key_version: "encryptionkey@0"
I20260620 07:14:07.903167 24030 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 07:14:07.906363 24044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:07.907603 24030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:14:07.907790 24030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "3c59cdc5715a45768fc27cfbaaab2fd7"
format_stamp: "Formatted at 2026-06-20 07:14:07 on dist-test-slave-4n6h"
server_key: "9db4269af69e2936ce97c13b41e1b92f"
server_key_iv: "099f3b50a885f9611343887df3b94c9b"
server_key_version: "encryptionkey@0"
I20260620 07:14:07.907992 24030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:07.922446 24030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:07.923193 24030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:07.923470 24030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:07.941190 24030 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:45199
I20260620 07:14:07.941212 24095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:45199 every 8 connection(s)
I20260620 07:14:07.942502 24030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:07.945876 24096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:07.946925 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 24030
I20260620 07:14:07.947158 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:07.947618 6357 external_mini_cluster.cc:1428] Setting key b79e0cb0dcb4031ce4bdeb116bcb9305
I20260620 07:14:07.959872 24096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7: Bootstrap starting.
I20260620 07:14:07.961999 24099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:07.949393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.6.53.126:45199)
0620 07:14:07.950550 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:07.950571 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:07.950619 (+ 48us) server_negotiation.cc:407] Connection header received
0620 07:14:07.952338 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:07.952363 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:07.952925 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:07.953344 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:07.953814 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:07.955631 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:07.957086 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:07.958102 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:07.958296 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:07.959409 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:07.959423 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:07.959439 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:07.960102 (+ 663us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:07.960194 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:07.960217 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:07.960330 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:07.960539 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 07:14:07.960822 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":384,"threads_started":1}
I20260620 07:14:07.963404 24096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:07.964397 24096 log.cc:824] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:07.967128 24096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7: No bootstrap required, opened a new log
I20260620 07:14:07.973883 24096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c59cdc5715a45768fc27cfbaaab2fd7" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45199 } }
I20260620 07:14:07.974198 24096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:07.974334 24096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c59cdc5715a45768fc27cfbaaab2fd7, State: Initialized, Role: FOLLOWER
I20260620 07:14:07.974774 24096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c59cdc5715a45768fc27cfbaaab2fd7" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45199 } }
I20260620 07:14:07.974965 24096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:07.975054 24096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:07.975170 24096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:07.977232 24096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c59cdc5715a45768fc27cfbaaab2fd7" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45199 } }
I20260620 07:14:07.977631 24096 leader_election.cc:302] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c59cdc5715a45768fc27cfbaaab2fd7; no voters:
I20260620 07:14:07.978202 24096 leader_election.cc:288] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:07.978334 24101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:07.979069 24101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [term 1 LEADER]: Becoming Leader. State: Replica: 3c59cdc5715a45768fc27cfbaaab2fd7, State: Running, Role: LEADER
I20260620 07:14:07.979657 24101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c59cdc5715a45768fc27cfbaaab2fd7" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45199 } }
I20260620 07:14:07.980317 24096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:07.982434 24102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c59cdc5715a45768fc27cfbaaab2fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c59cdc5715a45768fc27cfbaaab2fd7" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45199 } } }
I20260620 07:14:07.982539 24103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c59cdc5715a45768fc27cfbaaab2fd7. Latest consensus state: current_term: 1 leader_uuid: "3c59cdc5715a45768fc27cfbaaab2fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c59cdc5715a45768fc27cfbaaab2fd7" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 45199 } } }
I20260620 07:14:07.983017 24102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:07.983038 24103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:07.984784 24111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:07.989405 24111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:07.998279 24111 catalog_manager.cc:1347] Generated new cluster ID: a4e12797dd4d4dc3ab42e0abb9d14f16
I20260620 07:14:07.998383 24111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:08.018676 24111 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:08.020113 24111 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:08.030644 24111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7: Generated new TSK 0
I20260620 07:14:08.031411 24111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:08.051003 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:45199
--builtin_ntp_servers=127.6.53.84:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:14:08.389281 24120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:08.389540 24120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:08.389715 24120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:08.397158 24120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:08.397248 24120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:08.397424 24120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:14:08.406939 24120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:45199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:08.408213 24120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:08.409940 24120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:08.423249 24125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:08.423295 24126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:08.423318 24128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:08.427966 24120 server_base.cc:1034] running on GCE node
I20260620 07:14:08.428639 24120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:08.429610 24120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:08.430858 24120 hybrid_clock.cc:648] HybridClock initialized: now 1781939648430797 us; error 45 us; skew 500 ppm
I20260620 07:14:08.431203 24120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:14:08.433482 24120 webserver.cc:460] Webserver started at http://127.6.53.65:38103/ using document root <none> and password file <none>
I20260620 07:14:08.434207 24120 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:08.434409 24120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:08.434760 24120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:08.437956 24120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "8265b4b1abbc42959ab61b3db0a9f1b5"
format_stamp: "Formatted at 2026-06-20 07:14:08 on dist-test-slave-4n6h"
server_key: "801f3d7047267f3d73b6390ccab54c9a"
server_key_iv: "833ee565eebf17e9c68c19f8186da973"
server_key_version: "encryptionkey@0"
I20260620 07:14:08.438870 24120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "8265b4b1abbc42959ab61b3db0a9f1b5"
format_stamp: "Formatted at 2026-06-20 07:14:08 on dist-test-slave-4n6h"
server_key: "801f3d7047267f3d73b6390ccab54c9a"
server_key_iv: "833ee565eebf17e9c68c19f8186da973"
server_key_version: "encryptionkey@0"
I20260620 07:14:08.444115 24120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 07:14:08.447831 24134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:08.449098 24120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:14:08.449290 24120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "8265b4b1abbc42959ab61b3db0a9f1b5"
format_stamp: "Formatted at 2026-06-20 07:14:08 on dist-test-slave-4n6h"
server_key: "801f3d7047267f3d73b6390ccab54c9a"
server_key_iv: "833ee565eebf17e9c68c19f8186da973"
server_key_version: "encryptionkey@0"
I20260620 07:14:08.449528 24120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:08.482373 24120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:08.483269 24120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:08.483577 24120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:08.484714 24120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:08.486366 24120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:08.486490 24120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:08.486631 24120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:08.486737 24120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:08.520954 24120 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:37833
I20260620 07:14:08.521014 24246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:37833 every 8 connection(s)
I20260620 07:14:08.522375 24120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:14:08.528419 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 24120
I20260620 07:14:08.528661 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:14:08.529289 6357 external_mini_cluster.cc:1428] Setting key aa35175a6d0c5517599c1326e09f66b0
I20260620 07:14:08.532495 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:45199
--builtin_ntp_servers=127.6.53.84:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:14:08.537226 24249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:08.524974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:49275 (local address 127.6.53.126:45199)
0620 07:14:08.525504 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:08.525511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:08.526819 (+ 1308us) server_negotiation.cc:407] Connection header received
0620 07:14:08.528601 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:08.528607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:08.528705 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:08.528831 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:08.530375 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:08.531424 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:08.534175 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:08.534653 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:08.534750 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:08.536489 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:08.536493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:08.536496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:08.536515 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:08.536553 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:08.536564 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:08.536632 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:08.536909 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 07:14:08.536984 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":282,"threads_started":1}
I20260620 07:14:08.538885 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:08.525626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45199 (local address 127.6.53.65:49275)
0620 07:14:08.526662 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:08.526696 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:08.528337 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:08.529201 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:08.529215 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:08.529340 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:08.530148 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:08.530175 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:08.531579 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:08.531585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:08.533973 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:08.534003 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:08.534894 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:08.534904 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:08.535307 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:08.536120 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:14:08.536126 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:08.536182 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:14:08.536292 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:08.536667 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:08.536682 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:08.536693 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 07:14:08.536889 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 07:14:08.537348 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":291,"threads_started":1}
I20260620 07:14:08.540472 24247 heartbeater.cc:347] Connected to a master server at 127.6.53.126:45199
I20260620 07:14:08.540766 24247 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:08.541519 24247 heartbeater.cc:511] Master 127.6.53.126:45199 requested a full tablet report, sending...
I20260620 07:14:08.543643 24061 ts_manager.cc:194] Registered new tserver with Master: 8265b4b1abbc42959ab61b3db0a9f1b5 (127.6.53.65:37833)
I20260620 07:14:08.545188 24061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.53.65:49275
W20260620 07:14:08.850858 24251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:08.851131 24251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:08.851279 24251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:08.858667 24251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:08.858776 24251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:08.858955 24251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:14:08.867898 24251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:45199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:08.869154 24251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:08.870893 24251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:08.883718 24257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:08.883683 24259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:08.883919 24256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:08.887658 24251 server_base.cc:1034] running on GCE node
I20260620 07:14:08.888198 24251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:08.889050 24251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:08.890240 24251 hybrid_clock.cc:648] HybridClock initialized: now 1781939648890177 us; error 54 us; skew 500 ppm
I20260620 07:14:08.890549 24251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:14:08.892781 24251 webserver.cc:460] Webserver started at http://127.6.53.66:36385/ using document root <none> and password file <none>
I20260620 07:14:08.893427 24251 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:08.893509 24251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:08.893755 24251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:08.896603 24251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "b46be6e1aab649fca225e753e8f5f496"
format_stamp: "Formatted at 2026-06-20 07:14:08 on dist-test-slave-4n6h"
server_key: "2783d915b162ee80c5b0b1dc77780017"
server_key_iv: "9d314e79a8e26179f67ce8190c1b6f8a"
server_key_version: "encryptionkey@0"
I20260620 07:14:08.897312 24251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "b46be6e1aab649fca225e753e8f5f496"
format_stamp: "Formatted at 2026-06-20 07:14:08 on dist-test-slave-4n6h"
server_key: "2783d915b162ee80c5b0b1dc77780017"
server_key_iv: "9d314e79a8e26179f67ce8190c1b6f8a"
server_key_version: "encryptionkey@0"
I20260620 07:14:08.901975 24251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260620 07:14:08.905184 24265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:08.906544 24251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:08.906702 24251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "b46be6e1aab649fca225e753e8f5f496"
format_stamp: "Formatted at 2026-06-20 07:14:08 on dist-test-slave-4n6h"
server_key: "2783d915b162ee80c5b0b1dc77780017"
server_key_iv: "9d314e79a8e26179f67ce8190c1b6f8a"
server_key_version: "encryptionkey@0"
I20260620 07:14:08.906888 24251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:08.929159 24251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:08.929906 24251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:08.930133 24251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:08.931066 24251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:08.932502 24251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:08.932586 24251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:08.932653 24251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:08.932710 24251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:08.966331 24251 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:43359
I20260620 07:14:08.966426 24377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:43359 every 8 connection(s)
I20260620 07:14:08.967597 24251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:14:08.969455 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 24251
I20260620 07:14:08.969621 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:14:08.970150 6357 external_mini_cluster.cc:1428] Setting key 0da9f33f9b48c4aaef9a9bf65d522a3d
I20260620 07:14:08.976807 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:45199
--builtin_ntp_servers=127.6.53.84:33507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:14:08.980470 24249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:08.970077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:49461 (local address 127.6.53.126:45199)
0620 07:14:08.970345 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:08.970353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:08.971770 (+ 1417us) server_negotiation.cc:407] Connection header received
0620 07:14:08.973466 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:08.973472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:08.973566 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:08.973695 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:08.974745 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:08.975526 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:08.977306 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:08.978053 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:08.978189 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:08.979576 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:08.979585 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:08.979592 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:08.979619 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:08.979660 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:08.979678 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:08.979770 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:08.979975 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 07:14:08.980082 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:14:08.981415 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:08.970590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45199 (local address 127.6.53.66:49461)
0620 07:14:08.971592 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:08.971643 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:08.973219 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:08.973862 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:08.973869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:08.973953 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:08.974557 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:08.974574 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:08.975693 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:08.975698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:08.977159 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:08.977172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:08.978369 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:08.978375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:08.978572 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:08.979275 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:14:08.979278 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:08.979313 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:14:08.979388 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:08.979773 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:08.979780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:08.979786 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 07:14:08.979911 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 07:14:08.980302 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":334,"threads_started":1}
I20260620 07:14:08.982785 24378 heartbeater.cc:347] Connected to a master server at 127.6.53.126:45199
I20260620 07:14:08.983088 24378 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:08.983851 24378 heartbeater.cc:511] Master 127.6.53.126:45199 requested a full tablet report, sending...
I20260620 07:14:08.985422 24061 ts_manager.cc:194] Registered new tserver with Master: b46be6e1aab649fca225e753e8f5f496 (127.6.53.66:43359)
I20260620 07:14:08.986189 24061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.53.66:49461
W20260620 07:14:09.293638 24381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:09.293885 24381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:09.294046 24381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:09.301071 24381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:09.301180 24381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:09.301349 24381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:14:09.310113 24381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:33507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:45199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:09.311427 24381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:09.313127 24381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:09.325593 24386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:09.325625 24387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:09.325656 24389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:09.326596 24381 server_base.cc:1034] running on GCE node
I20260620 07:14:09.327203 24381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:09.328112 24381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:09.329330 24381 hybrid_clock.cc:648] HybridClock initialized: now 1781939649329293 us; error 52 us; skew 500 ppm
I20260620 07:14:09.329630 24381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:14:09.331768 24381 webserver.cc:460] Webserver started at http://127.6.53.67:38549/ using document root <none> and password file <none>
I20260620 07:14:09.332422 24381 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:09.332549 24381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:09.332860 24381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:09.335588 24381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "d57c86e663104d83a00cd35f938fd6e1"
format_stamp: "Formatted at 2026-06-20 07:14:09 on dist-test-slave-4n6h"
server_key: "bd42c923bb99b950f34f63258ecf5450"
server_key_iv: "54330de92b89e2f7cbcfd4a37e344a61"
server_key_version: "encryptionkey@0"
I20260620 07:14:09.336346 24381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "d57c86e663104d83a00cd35f938fd6e1"
format_stamp: "Formatted at 2026-06-20 07:14:09 on dist-test-slave-4n6h"
server_key: "bd42c923bb99b950f34f63258ecf5450"
server_key_iv: "54330de92b89e2f7cbcfd4a37e344a61"
server_key_version: "encryptionkey@0"
I20260620 07:14:09.341172 24381 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260620 07:14:09.344655 24395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:09.345813 24381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 07:14:09.345979 24381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "d57c86e663104d83a00cd35f938fd6e1"
format_stamp: "Formatted at 2026-06-20 07:14:09 on dist-test-slave-4n6h"
server_key: "bd42c923bb99b950f34f63258ecf5450"
server_key_iv: "54330de92b89e2f7cbcfd4a37e344a61"
server_key_version: "encryptionkey@0"
I20260620 07:14:09.346190 24381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:09.360046 24381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:09.360862 24381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:09.361150 24381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:09.362126 24381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:09.363657 24381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:09.363781 24381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:09.363914 24381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:09.364002 24381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:09.397754 24381 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:43121
I20260620 07:14:09.397805 24507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:43121 every 8 connection(s)
I20260620 07:14:09.399042 24381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:14:09.400480 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 24381
I20260620 07:14:09.400661 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:14:09.401094 6357 external_mini_cluster.cc:1428] Setting key 9768e30991b3937ad965490fa4e57e7a
I20260620 07:14:09.411936 24249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.401520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:51861 (local address 127.6.53.126:45199)
0620 07:14:09.401752 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:09.401758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:09.403286 (+ 1528us) server_negotiation.cc:407] Connection header received
0620 07:14:09.405478 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:09.405482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:09.405544 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:09.405637 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:09.406828 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.407569 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.409152 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.409565 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.409658 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.411263 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:09.411267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:09.411271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:09.411288 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:09.411319 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:09.411330 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:09.411404 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:09.411622 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 07:14:09.411692 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:14:09.412981 24510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.402056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:45199 (local address 127.6.53.67:51861)
0620 07:14:09.403135 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:09.403176 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:09.405234 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:09.405852 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:09.405859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:09.405938 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:09.406640 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:09.406658 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.407718 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.407722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:09.409001 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:09.409026 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.409760 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.409765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:09.410116 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.410817 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:14:09.410820 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:09.410856 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:14:09.410996 (+ 140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:09.411423 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:09.411430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:09.411436 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 07:14:09.411584 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 07:14:09.411911 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":346,"threads_started":1}
I20260620 07:14:09.414448 24508 heartbeater.cc:347] Connected to a master server at 127.6.53.126:45199
I20260620 07:14:09.414736 24508 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:09.415418 24508 heartbeater.cc:511] Master 127.6.53.126:45199 requested a full tablet report, sending...
I20260620 07:14:09.416756 24061 ts_manager.cc:194] Registered new tserver with Master: d57c86e663104d83a00cd35f938fd6e1 (127.6.53.67:43121)
I20260620 07:14:09.417603 24061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.53.67:51861
I20260620 07:14:09.428684 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:14:09.438436 24249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.432788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59478 (local address 127.6.53.126:45199)
0620 07:14:09.432967 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:09.432971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:09.433386 (+ 415us) server_negotiation.cc:407] Connection header received
0620 07:14:09.433521 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:09.433525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:09.433602 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:09.433686 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:09.434187 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.435008 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.436313 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.436707 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.436809 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.437793 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:09.437797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:09.437800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:09.437818 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:09.437846 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:09.437856 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:09.437922 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:09.438038 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 07:14:09.438106 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:14:09.442028 24061 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:14:09.448817 24061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:14:09.486531 24523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.461345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49546 (local address 127.6.53.66:43359)
0620 07:14:09.478444 (+ 17099us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:09.478453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:09.478485 (+ 32us) server_negotiation.cc:407] Connection header received
0620 07:14:09.478577 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:09.478586 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:09.478798 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:09.478987 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:09.479998 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.481607 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.484240 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.485370 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.485525 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.485751 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:09.485999 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 07:14:09.486173 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16915,"thread_start_us":17152,"threads_started":1}
I20260620 07:14:09.486627 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.460659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:43359 (local address 127.0.0.1:49546)
0620 07:14:09.476532 (+ 15873us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:09.476557 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:09.476696 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:09.479122 (+ 2426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:09.479160 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:09.479188 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:09.479800 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:09.479817 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.481751 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.481756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:09.484019 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:09.484031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.485557 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.485562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:09.485756 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.485822 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 07:14:09.485962 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 07:14:09.486148 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15689,"mutex_wait_us":140,"thread_start_us":12313,"threads_started":1}
I20260620 07:14:09.486804 24521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.460509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:43121 (local address 127.0.0.1:41446)
0620 07:14:09.473454 (+ 12945us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:09.473513 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:09.473744 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:09.478393 (+ 4649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:09.478400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:09.478447 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:09.479318 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:09.479335 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.481055 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.481061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:09.483693 (+ 2632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:09.483708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.485820 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.485826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:09.486043 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.486085 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 07:14:09.486160 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 07:14:09.486321 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12621,"mutex_wait_us":25,"spinlock_wait_cycles":896,"thread_start_us":16288,"threads_started":1}
I20260620 07:14:09.487534 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.460812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41446 (local address 127.6.53.67:43121)
0620 07:14:09.472820 (+ 12008us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:09.472829 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:09.473972 (+ 1143us) server_negotiation.cc:407] Connection header received
0620 07:14:09.477852 (+ 3880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:09.477863 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:09.478142 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:09.478360 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:09.479542 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.480918 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.483911 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.485267 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.485368 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.485648 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:09.487004 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0620 07:14:09.487157 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11811,"thread_start_us":11663,"threads_started":1}
I20260620 07:14:09.487892 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.474229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.6.53.65:37833)
0620 07:14:09.477872 (+ 3643us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:09.477885 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:09.478629 (+ 744us) server_negotiation.cc:407] Connection header received
0620 07:14:09.478730 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:09.478737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:09.478923 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:09.479083 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:09.480111 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.481609 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.484603 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.485987 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.486107 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.486336 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:09.487391 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0620 07:14:09.487562 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3443,"thread_start_us":3270,"threads_started":1}
I20260620 07:14:09.487278 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.474232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37833 (local address 127.0.0.1:59314)
0620 07:14:09.478302 (+ 4070us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:09.478319 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:09.478424 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:09.479327 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:09.479330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:09.479353 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:09.479929 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:09.479943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.482380 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.482385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:09.484419 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:09.484433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.486457 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.486466 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:09.486665 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.486703 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 07:14:09.486833 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 07:14:09.486971 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3937,"mutex_wait_us":48,"thread_start_us":3789,"threads_started":1}
I20260620 07:14:09.490624 24313 tablet_service.cc:1467] Processing CreateTablet for tablet 89e315f79ce544b2aa98ccb3719fc731 (DEFAULT_TABLE table=test-table [id=cc711a7e1b0f4179ad84e22e9cc92cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:09.491310 24182 tablet_service.cc:1467] Processing CreateTablet for tablet 89e315f79ce544b2aa98ccb3719fc731 (DEFAULT_TABLE table=test-table [id=cc711a7e1b0f4179ad84e22e9cc92cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:09.492034 24313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89e315f79ce544b2aa98ccb3719fc731. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:09.492550 24182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89e315f79ce544b2aa98ccb3719fc731. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:09.497001 24443 tablet_service.cc:1467] Processing CreateTablet for tablet 89e315f79ce544b2aa98ccb3719fc731 (DEFAULT_TABLE table=test-table [id=cc711a7e1b0f4179ad84e22e9cc92cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:09.498044 24443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89e315f79ce544b2aa98ccb3719fc731. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:09.506559 24527 tablet_bootstrap.cc:492] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5: Bootstrap starting.
I20260620 07:14:09.507344 24526 tablet_bootstrap.cc:492] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496: Bootstrap starting.
I20260620 07:14:09.508528 24528 tablet_bootstrap.cc:492] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1: Bootstrap starting.
I20260620 07:14:09.511750 24528 tablet_bootstrap.cc:654] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:09.511802 24527 tablet_bootstrap.cc:654] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:09.512794 24528 log.cc:824] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:09.512809 24527 log.cc:824] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:09.515225 24526 tablet_bootstrap.cc:654] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:09.515502 24528 tablet_bootstrap.cc:492] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1: No bootstrap required, opened a new log
I20260620 07:14:09.515748 24528 ts_tablet_manager.cc:1397] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 07:14:09.516162 24526 log.cc:824] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:09.518746 24527 tablet_bootstrap.cc:492] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5: No bootstrap required, opened a new log
I20260620 07:14:09.518834 24526 tablet_bootstrap.cc:492] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496: No bootstrap required, opened a new log
I20260620 07:14:09.519004 24527 ts_tablet_manager.cc:1397] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20260620 07:14:09.519090 24526 ts_tablet_manager.cc:1397] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260620 07:14:09.523833 24528 raft_consensus.cc:348] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57c86e663104d83a00cd35f938fd6e1" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 43121 } } peers { permanent_uuid: "b46be6e1aab649fca225e753e8f5f496" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43359 } } peers { permanent_uuid: "8265b4b1abbc42959ab61b3db0a9f1b5" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37833 } }
I20260620 07:14:09.524173 24528 raft_consensus.cc:374] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:09.524317 24528 raft_consensus.cc:729] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57c86e663104d83a00cd35f938fd6e1, State: Initialized, Role: FOLLOWER
I20260620 07:14:09.525147 24528 consensus_queue.cc:260] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57c86e663104d83a00cd35f938fd6e1" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 43121 } } peers { permanent_uuid: "b46be6e1aab649fca225e753e8f5f496" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43359 } } peers { permanent_uuid: "8265b4b1abbc42959ab61b3db0a9f1b5" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37833 } }
I20260620 07:14:09.529147 24526 raft_consensus.cc:348] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57c86e663104d83a00cd35f938fd6e1" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 43121 } } peers { permanent_uuid: "b46be6e1aab649fca225e753e8f5f496" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43359 } } peers { permanent_uuid: "8265b4b1abbc42959ab61b3db0a9f1b5" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37833 } }
I20260620 07:14:09.529577 24526 raft_consensus.cc:374] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:09.529729 24526 raft_consensus.cc:729] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b46be6e1aab649fca225e753e8f5f496, State: Initialized, Role: FOLLOWER
I20260620 07:14:09.530390 24526 consensus_queue.cc:260] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57c86e663104d83a00cd35f938fd6e1" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 43121 } } peers { permanent_uuid: "b46be6e1aab649fca225e753e8f5f496" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43359 } } peers { permanent_uuid: "8265b4b1abbc42959ab61b3db0a9f1b5" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37833 } }
I20260620 07:14:09.533306 24528 ts_tablet_manager.cc:1428] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20260620 07:14:09.533682 24508 heartbeater.cc:503] Master 127.6.53.126:45199 was elected leader, sending a full tablet report...
I20260620 07:14:09.535779 24378 heartbeater.cc:503] Master 127.6.53.126:45199 was elected leader, sending a full tablet report...
I20260620 07:14:09.536043 24527 raft_consensus.cc:348] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57c86e663104d83a00cd35f938fd6e1" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 43121 } } peers { permanent_uuid: "b46be6e1aab649fca225e753e8f5f496" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43359 } } peers { permanent_uuid: "8265b4b1abbc42959ab61b3db0a9f1b5" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37833 } }
I20260620 07:14:09.536445 24527 raft_consensus.cc:374] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:09.536547 24527 raft_consensus.cc:729] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8265b4b1abbc42959ab61b3db0a9f1b5, State: Initialized, Role: FOLLOWER
I20260620 07:14:09.537441 24527 consensus_queue.cc:260] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57c86e663104d83a00cd35f938fd6e1" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 43121 } } peers { permanent_uuid: "b46be6e1aab649fca225e753e8f5f496" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43359 } } peers { permanent_uuid: "8265b4b1abbc42959ab61b3db0a9f1b5" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37833 } }
I20260620 07:14:09.538183 24526 ts_tablet_manager.cc:1428] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496: Time spent starting tablet: real 0.019s user 0.012s sys 0.004s
I20260620 07:14:09.539819 24247 heartbeater.cc:503] Master 127.6.53.126:45199 was elected leader, sending a full tablet report...
I20260620 07:14:09.540518 24527 ts_tablet_manager.cc:1428] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5: Time spent starting tablet: real 0.021s user 0.013s sys 0.002s
I20260620 07:14:09.615353 24534 raft_consensus.cc:482] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:09.615703 24534 raft_consensus.cc:504] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57c86e663104d83a00cd35f938fd6e1" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 43121 } } peers { permanent_uuid: "b46be6e1aab649fca225e753e8f5f496" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43359 } } peers { permanent_uuid: "8265b4b1abbc42959ab61b3db0a9f1b5" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37833 } }
I20260620 07:14:09.617355 24534 leader_election.cc:288] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d57c86e663104d83a00cd35f938fd6e1 (127.6.53.67:43121), b46be6e1aab649fca225e753e8f5f496 (127.6.53.66:43359)
I20260620 07:14:09.624121 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.617726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:34157 (local address 127.6.53.67:43121)
0620 07:14:09.617962 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:09.617971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:09.618645 (+ 674us) server_negotiation.cc:407] Connection header received
0620 07:14:09.618867 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:09.618870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:09.618929 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:09.619026 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:09.619682 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.620432 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.622421 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.623203 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.623302 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.623433 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:09.623747 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 07:14:09.623881 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:14:09.624166 24535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.617548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:43121 (local address 127.6.53.65:34157)
0620 07:14:09.618519 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:09.618534 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:09.618710 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:09.619043 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:09.619047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:09.619070 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:09.619495 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:09.619518 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.620569 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.620575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:09.622209 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:09.622222 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.623399 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.623405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:09.623591 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.623617 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 07:14:09.623687 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 07:14:09.623811 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":33,"thread_start_us":628,"threads_started":1}
I20260620 07:14:09.624261 24523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.617869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:35127 (local address 127.6.53.66:43359)
0620 07:14:09.618100 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:09.618109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:09.618812 (+ 703us) server_negotiation.cc:407] Connection header received
0620 07:14:09.619009 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:09.619013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:09.619069 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:09.619166 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:09.619867 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.620569 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.622490 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.623209 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.623387 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.623476 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:09.623863 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 07:14:09.623996 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:14:09.624305 24536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.617808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:43359 (local address 127.6.53.65:35127)
0620 07:14:09.618698 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:09.618711 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:09.618861 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:09.619184 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:09.619188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:09.619226 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:09.619702 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:09.619715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.620703 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.620706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:09.622226 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:09.622237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.623443 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.623451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:09.623654 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.623714 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 07:14:09.623803 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 07:14:09.623911 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":44,"thread_start_us":553,"threads_started":1}
I20260620 07:14:09.625247 24333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89e315f79ce544b2aa98ccb3719fc731" candidate_uuid: "8265b4b1abbc42959ab61b3db0a9f1b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b46be6e1aab649fca225e753e8f5f496" is_pre_election: true
I20260620 07:14:09.625588 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89e315f79ce544b2aa98ccb3719fc731" candidate_uuid: "8265b4b1abbc42959ab61b3db0a9f1b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57c86e663104d83a00cd35f938fd6e1" is_pre_election: true
I20260620 07:14:09.625869 24333 raft_consensus.cc:2454] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8265b4b1abbc42959ab61b3db0a9f1b5 in term 0.
I20260620 07:14:09.626122 24463 raft_consensus.cc:2454] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8265b4b1abbc42959ab61b3db0a9f1b5 in term 0.
I20260620 07:14:09.626694 24136 leader_election.cc:302] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8265b4b1abbc42959ab61b3db0a9f1b5, b46be6e1aab649fca225e753e8f5f496; no voters:
I20260620 07:14:09.627163 24534 raft_consensus.cc:2781] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:14:09.627321 24534 raft_consensus.cc:482] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:14:09.627426 24534 raft_consensus.cc:3037] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:09.629493 24534 raft_consensus.cc:504] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57c86e663104d83a00cd35f938fd6e1" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 43121 } } peers { permanent_uuid: "b46be6e1aab649fca225e753e8f5f496" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43359 } } peers { permanent_uuid: "8265b4b1abbc42959ab61b3db0a9f1b5" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37833 } }
I20260620 07:14:09.630148 24534 leader_election.cc:288] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [CANDIDATE]: Term 1 election: Requested vote from peers d57c86e663104d83a00cd35f938fd6e1 (127.6.53.67:43121), b46be6e1aab649fca225e753e8f5f496 (127.6.53.66:43359)
I20260620 07:14:09.630641 24333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89e315f79ce544b2aa98ccb3719fc731" candidate_uuid: "8265b4b1abbc42959ab61b3db0a9f1b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b46be6e1aab649fca225e753e8f5f496"
I20260620 07:14:09.630641 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89e315f79ce544b2aa98ccb3719fc731" candidate_uuid: "8265b4b1abbc42959ab61b3db0a9f1b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57c86e663104d83a00cd35f938fd6e1"
I20260620 07:14:09.630859 24333 raft_consensus.cc:3037] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:09.630858 24463 raft_consensus.cc:3037] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:09.634136 24333 raft_consensus.cc:2454] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8265b4b1abbc42959ab61b3db0a9f1b5 in term 1.
I20260620 07:14:09.634131 24463 raft_consensus.cc:2454] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8265b4b1abbc42959ab61b3db0a9f1b5 in term 1.
I20260620 07:14:09.634594 24136 leader_election.cc:302] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8265b4b1abbc42959ab61b3db0a9f1b5, b46be6e1aab649fca225e753e8f5f496, d57c86e663104d83a00cd35f938fd6e1; no voters:
I20260620 07:14:09.634907 24534 raft_consensus.cc:2781] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:09.635346 24534 raft_consensus.cc:686] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [term 1 LEADER]: Becoming Leader. State: Replica: 8265b4b1abbc42959ab61b3db0a9f1b5, State: Running, Role: LEADER
I20260620 07:14:09.635800 24534 consensus_queue.cc:237] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57c86e663104d83a00cd35f938fd6e1" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 43121 } } peers { permanent_uuid: "b46be6e1aab649fca225e753e8f5f496" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43359 } } peers { permanent_uuid: "8265b4b1abbc42959ab61b3db0a9f1b5" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37833 } }
I20260620 07:14:09.640019 24060 catalog_manager.cc:5495] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8265b4b1abbc42959ab61b3db0a9f1b5 (127.6.53.65). New cstate: current_term: 1 leader_uuid: "8265b4b1abbc42959ab61b3db0a9f1b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57c86e663104d83a00cd35f938fd6e1" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 43121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b46be6e1aab649fca225e753e8f5f496" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8265b4b1abbc42959ab61b3db0a9f1b5" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37833 } health_report { overall_health: HEALTHY } } }
W20260620 07:14:09.650573 24509 tablet.cc:2369] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:14:09.660818 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.655725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.6.53.65:37833)
0620 07:14:09.655893 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:09.655897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:09.655914 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:14:09.656000 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:09.656003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:09.656081 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:09.656190 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:14:09.656492 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.657163 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.658685 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.659095 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.659185 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.659670 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:14:09.660139 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:14:09.660348 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:09.660411 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 07:14:09.660527 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:14:09.669368 24463 raft_consensus.cc:1261] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1 [term 1 FOLLOWER]: Refusing update from remote peer 8265b4b1abbc42959ab61b3db0a9f1b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:09.669370 24333 raft_consensus.cc:1261] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496 [term 1 FOLLOWER]: Refusing update from remote peer 8265b4b1abbc42959ab61b3db0a9f1b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:09.670452 24537 consensus_queue.cc:1035] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d57c86e663104d83a00cd35f938fd6e1" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 43121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:09.670856 24534 consensus_queue.cc:1035] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b46be6e1aab649fca225e753e8f5f496" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 43359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:09.697484 24061 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:09.697741 24061 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:09.701367 24061 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c59cdc5715a45768fc27cfbaaab2fd7: Sending DeleteTablet for 3 replicas of tablet 89e315f79ce544b2aa98ccb3719fc731
I20260620 07:14:09.702539 24313 tablet_service.cc:1514] Processing DeleteTablet for tablet 89e315f79ce544b2aa98ccb3719fc731 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:09 UTC) from {username='slave'} at 127.0.0.1:49546
I20260620 07:14:09.702539 24443 tablet_service.cc:1514] Processing DeleteTablet for tablet 89e315f79ce544b2aa98ccb3719fc731 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:09 UTC) from {username='slave'} at 127.0.0.1:41446
I20260620 07:14:09.702726 24182 tablet_service.cc:1514] Processing DeleteTablet for tablet 89e315f79ce544b2aa98ccb3719fc731 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:09 UTC) from {username='slave'} at 127.0.0.1:59314
I20260620 07:14:09.709674 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 24120
I20260620 07:14:09.719184 24552 tablet_replica.cc:330] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5: stopping tablet replica
I20260620 07:14:09.720966 24551 tablet_replica.cc:330] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496: stopping tablet replica
I20260620 07:14:09.721033 24552 raft_consensus.cc:2229] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:14:09.721385 24551 raft_consensus.cc:2229] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:09.721839 24552 raft_consensus.cc:2258] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:09.721892 24551 raft_consensus.cc:2258] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:09.722362 24550 tablet_replica.cc:330] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1: stopping tablet replica
I20260620 07:14:09.722774 24550 raft_consensus.cc:2229] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:09.723358 24550 raft_consensus.cc:2258] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:09.723701 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.711323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.6.53.65:37833)
0620 07:14:09.711547 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:09.711556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:09.712654 (+ 1098us) server_negotiation.cc:407] Connection header received
0620 07:14:09.713504 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:09.713513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:09.713640 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:09.713743 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:09.715164 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.716765 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.719766 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.720383 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.720534 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.721964 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:09.721974 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:09.721989 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:09.722835 (+ 846us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:09.722888 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:09.722910 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:09.723000 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:09.723245 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 07:14:09.723341 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:14:09.724349 24552 ts_tablet_manager.cc:1905] T 89e315f79ce544b2aa98ccb3719fc731 P 8265b4b1abbc42959ab61b3db0a9f1b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:09.725266 24242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:09.725591 24551 ts_tablet_manager.cc:1905] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:09.727198 24550 ts_tablet_manager.cc:1905] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:09.732829 24551 ts_tablet_manager.cc:1918] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:09.732996 24551 log.cc:1196] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/89e315f79ce544b2aa98ccb3719fc731
I20260620 07:14:09.733573 24551 ts_tablet_manager.cc:1939] T 89e315f79ce544b2aa98ccb3719fc731 P b46be6e1aab649fca225e753e8f5f496: Deleting consensus metadata
I20260620 07:14:09.735265 24550 ts_tablet_manager.cc:1918] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:09.735435 24550 log.cc:1196] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/89e315f79ce544b2aa98ccb3719fc731
I20260620 07:14:09.735667 24046 catalog_manager.cc:4841] TS b46be6e1aab649fca225e753e8f5f496 (127.6.53.66:43359): tablet 89e315f79ce544b2aa98ccb3719fc731 (table test-table [id=cc711a7e1b0f4179ad84e22e9cc92cd3]) successfully deleted
I20260620 07:14:09.736001 24550 ts_tablet_manager.cc:1939] T 89e315f79ce544b2aa98ccb3719fc731 P d57c86e663104d83a00cd35f938fd6e1: Deleting consensus metadata
I20260620 07:14:09.737531 24048 catalog_manager.cc:4841] TS d57c86e663104d83a00cd35f938fd6e1 (127.6.53.67:43121): tablet 89e315f79ce544b2aa98ccb3719fc731 (table test-table [id=cc711a7e1b0f4179ad84e22e9cc92cd3]) successfully deleted
I20260620 07:14:09.824375 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 24120
W20260620 07:14:09.846110 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.53.65:37833 (error 108)
I20260620 07:14:09.846220 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 24251
I20260620 07:14:09.847879 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.847422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37833 (local address 127.0.0.1:59344)
0620 07:14:09.847602 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:09.847710 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 07:14:09.848389 24046 catalog_manager.cc:4584] TS 8265b4b1abbc42959ab61b3db0a9f1b5 (127.6.53.65:37833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 89e315f79ce544b2aa98ccb3719fc731: Network error: Client connection negotiation failed: client connection to 127.6.53.65:37833: connect: Connection refused (error 111)
I20260620 07:14:09.853554 24523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.847432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.6.53.66:43359)
0620 07:14:09.847587 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:09.847591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:09.847605 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:14:09.847658 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:09.847661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:09.847751 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:09.847829 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:09.848290 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.849126 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.850597 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.851031 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.851145 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.852158 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:09.852166 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:09.852179 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:09.852755 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:09.852810 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:09.852836 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:09.852938 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:09.853072 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 07:14:09.853164 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:14:09.854532 24373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:09.901985 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.901543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37833 (local address 127.0.0.1:59352)
0620 07:14:09.901733 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:09.901822 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 07:14:09.902983 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.902675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37833 (local address 127.0.0.1:59368)
0620 07:14:09.902803 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:09.902882 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 07:14:09.942528 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 24251
I20260620 07:14:09.963038 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 24381
I20260620 07:14:09.963963 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.963421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37833 (local address 127.0.0.1:59374)
0620 07:14:09.963669 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:09.963766 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 07:14:09.965004 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.964643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37833 (local address 127.0.0.1:59388)
0620 07:14:09.964829 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:09.964899 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 07:14:09.970781 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.963862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.6.53.67:43121)
0620 07:14:09.964046 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:09.964053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:09.964237 (+ 184us) server_negotiation.cc:407] Connection header received
0620 07:14:09.964378 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:09.964383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:09.964502 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:09.964620 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:09.965138 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.965894 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.967954 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.968347 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.968439 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.969580 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:09.969589 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:09.969602 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:09.970142 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:09.970185 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:09.970203 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:09.970356 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:09.970491 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 07:14:09.970571 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:14:09.971448 24503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:10.045804 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:10.045238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37833 (local address 127.0.0.1:59402)
0620 07:14:10.045500 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:10.045614 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 07:14:10.047011 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:10.046538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37833 (local address 127.0.0.1:59414)
0620 07:14:10.046780 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:10.046875 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 07:14:10.062371 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 24381
I20260620 07:14:10.078799 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 24030
I20260620 07:14:10.079883 24091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:10.188501 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 24030
2026-06-20T07:14:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2733 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:10 dist-test-slave-4n6h krb5kdc[24561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:10 dist-test-slave-4n6h krb5kdc[24561](info): set up 2 sockets
Jun 20 07:14:10 dist-test-slave-4n6h krb5kdc[24561](info): commencing operation
krb5kdc: starting...
W20260620 07:14:11.278306 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:11 dist-test-slave-4n6h krb5kdc[24561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:14:11.515578 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:40833
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:40833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:11.843467 24574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:11.843906 24574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:11.844090 24574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:11.851763 24574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:11.851883 24574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:11.851966 24574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:11.852111 24574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:11.852180 24574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:11.862133 24574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:40833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:40833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:11.863422 24574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:11.864682 24574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:11.876369 24579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:11.876223 24580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:11.876320 24582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:11.876969 24574 server_base.cc:1034] running on GCE node
I20260620 07:14:11.877714 24574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:11.879173 24574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:11.880424 24574 hybrid_clock.cc:648] HybridClock initialized: now 1781939651880340 us; error 70 us; skew 500 ppm
I20260620 07:14:11.880774 24574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:11 dist-test-slave-4n6h krb5kdc[24561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:11.883808 24574 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:14:11.885306 24574 webserver.cc:460] Webserver started at http://127.6.53.126:41203/ using document root <none> and password file <none>
I20260620 07:14:11.885946 24574 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:11.886087 24574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:11.886456 24574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:11.889298 24574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "aadb8d787ad84e7d94ba81161534267e"
format_stamp: "Formatted at 2026-06-20 07:14:11 on dist-test-slave-4n6h"
server_key: "aac9d3070bbabf3f799cca6ef64d1cbf"
server_key_iv: "4e6e9fd475e44fc66dc2084a2721daa8"
server_key_version: "encryptionkey@0"
I20260620 07:14:11.890076 24574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "aadb8d787ad84e7d94ba81161534267e"
format_stamp: "Formatted at 2026-06-20 07:14:11 on dist-test-slave-4n6h"
server_key: "aac9d3070bbabf3f799cca6ef64d1cbf"
server_key_iv: "4e6e9fd475e44fc66dc2084a2721daa8"
server_key_version: "encryptionkey@0"
I20260620 07:14:11.894830 24574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 07:14:11.897997 24589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:11.899191 24574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:14:11.899369 24574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "aadb8d787ad84e7d94ba81161534267e"
format_stamp: "Formatted at 2026-06-20 07:14:11 on dist-test-slave-4n6h"
server_key: "aac9d3070bbabf3f799cca6ef64d1cbf"
server_key_iv: "4e6e9fd475e44fc66dc2084a2721daa8"
server_key_version: "encryptionkey@0"
I20260620 07:14:11.899551 24574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:11.916318 24574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:11.920554 24574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:11.920850 24574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:11.940433 24574 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:40833
I20260620 07:14:11.940455 24640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:40833 every 8 connection(s)
I20260620 07:14:11.941927 24574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:11.945672 24641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:11.947588 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 24574
I20260620 07:14:11.947800 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:11.948315 6357 external_mini_cluster.cc:1428] Setting key 80e3f92d2190951553b6e044dc673695
I20260620 07:14:11.958818 24641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e: Bootstrap starting.
I20260620 07:14:11.962112 24641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:11.963233 24641 log.cc:824] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:14:11 dist-test-slave-4n6h krb5kdc[24561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:14:11.966187 24641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e: No bootstrap required, opened a new log
I20260620 07:14:11.972959 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:11.950234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.6.53.126:40833)
0620 07:14:11.951124 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:11.951139 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:11.951187 (+ 48us) server_negotiation.cc:407] Connection header received
0620 07:14:11.953473 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:11.953498 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:11.953946 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:11.954379 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:11.956236 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:11.958192 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:11.959753 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:11.960866 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:11.961034 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:11.965726 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:11.965734 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:11.965748 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:11.965782 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:11.968891 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:11.969740 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:11.969746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:11.969751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:11.969944 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:11.970378 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:11.970382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:11.970383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:11.970893 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:11.971102 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:11.971573 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 07:14:11.972018 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":356,"threads_started":1}
I20260620 07:14:11.974675 24641 raft_consensus.cc:348] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadb8d787ad84e7d94ba81161534267e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40833 } }
I20260620 07:14:11.975030 24641 raft_consensus.cc:374] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:11.975145 24641 raft_consensus.cc:729] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aadb8d787ad84e7d94ba81161534267e, State: Initialized, Role: FOLLOWER
I20260620 07:14:11.975725 24641 consensus_queue.cc:260] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadb8d787ad84e7d94ba81161534267e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40833 } }
I20260620 07:14:11.975946 24641 raft_consensus.cc:388] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:11.976076 24641 raft_consensus.cc:482] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:11.976222 24641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:11.978371 24641 raft_consensus.cc:504] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadb8d787ad84e7d94ba81161534267e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40833 } }
I20260620 07:14:11.978812 24641 leader_election.cc:302] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aadb8d787ad84e7d94ba81161534267e; no voters:
I20260620 07:14:11.979370 24641 leader_election.cc:288] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:11.979485 24646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:11.980171 24646 raft_consensus.cc:686] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [term 1 LEADER]: Becoming Leader. State: Replica: aadb8d787ad84e7d94ba81161534267e, State: Running, Role: LEADER
I20260620 07:14:11.980724 24646 consensus_queue.cc:237] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadb8d787ad84e7d94ba81161534267e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40833 } }
I20260620 07:14:11.981209 24641 sys_catalog.cc:564] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:11.983237 24648 sys_catalog.cc:455] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [sys.catalog]: SysCatalogTable state changed. Reason: New leader aadb8d787ad84e7d94ba81161534267e. Latest consensus state: current_term: 1 leader_uuid: "aadb8d787ad84e7d94ba81161534267e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadb8d787ad84e7d94ba81161534267e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40833 } } }
I20260620 07:14:11.983350 24647 sys_catalog.cc:455] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aadb8d787ad84e7d94ba81161534267e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadb8d787ad84e7d94ba81161534267e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40833 } } }
I20260620 07:14:11.983688 24648 sys_catalog.cc:458] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:11.986346 24647 sys_catalog.cc:458] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:11.986889 24652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:11.991286 24652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:11.999050 24652 catalog_manager.cc:1347] Generated new cluster ID: 3f50e76e115c4b03b5319ee9c6885a64
I20260620 07:14:11.999179 24652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:12.008332 24652 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:12.009593 24652 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:12.018162 24652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e: Generated new TSK 0
I20260620 07:14:12.018955 24652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:14:12.077872 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40833
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:12.406665 24667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:12.406978 24667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:12.407174 24667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:12.415237 24667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:12.415360 24667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:12.415544 24667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:14:12.425908 24667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:12.427225 24667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:12.428516 24667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:12.441111 24673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:12.441704 24675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:12.441862 24667 server_base.cc:1034] running on GCE node
W20260620 07:14:12.441946 24672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:12.442653 24667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:12.443554 24667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:12.444775 24667 hybrid_clock.cc:648] HybridClock initialized: now 1781939652444719 us; error 53 us; skew 500 ppm
I20260620 07:14:12.445113 24667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:12 dist-test-slave-4n6h krb5kdc[24561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:12.448402 24667 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:14:12.449966 24667 webserver.cc:460] Webserver started at http://127.6.53.65:38845/ using document root <none> and password file <none>
I20260620 07:14:12.450764 24667 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:12.450893 24667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:12.451210 24667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:12.454046 24667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "a13b3ab381d643828a29a4558fa7527c"
format_stamp: "Formatted at 2026-06-20 07:14:12 on dist-test-slave-4n6h"
server_key: "b7077f56cf4367f88ba7bd220e9d19d1"
server_key_iv: "cd59e1aaae92814be681a841564aca7e"
server_key_version: "encryptionkey@0"
I20260620 07:14:12.454861 24667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "a13b3ab381d643828a29a4558fa7527c"
format_stamp: "Formatted at 2026-06-20 07:14:12 on dist-test-slave-4n6h"
server_key: "b7077f56cf4367f88ba7bd220e9d19d1"
server_key_iv: "cd59e1aaae92814be681a841564aca7e"
server_key_version: "encryptionkey@0"
I20260620 07:14:12.459810 24667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:14:12.463102 24682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:12.464339 24667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:14:12.464540 24667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "a13b3ab381d643828a29a4558fa7527c"
format_stamp: "Formatted at 2026-06-20 07:14:12 on dist-test-slave-4n6h"
server_key: "b7077f56cf4367f88ba7bd220e9d19d1"
server_key_iv: "cd59e1aaae92814be681a841564aca7e"
server_key_version: "encryptionkey@0"
I20260620 07:14:12.464727 24667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:12.479883 24667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:12.484268 24667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:12.484596 24667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:12.485628 24667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:12.487232 24667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:12.487337 24667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:12.487480 24667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:12.487569 24667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:12.520177 24667 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:44883
I20260620 07:14:12.520195 24794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:44883 every 8 connection(s)
I20260620 07:14:12.521500 24667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:14:12.521847 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 24667
I20260620 07:14:12.522073 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:14:12.522713 6357 external_mini_cluster.cc:1428] Setting key 9d2d557ce5694dd2a18d970824b733fb
Jun 20 07:14:12 dist-test-slave-4n6h krb5kdc[24561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:14:12.544641 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:12.524164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:45847 (local address 127.6.53.126:40833)
0620 07:14:12.524621 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:12.524627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:12.525615 (+ 988us) server_negotiation.cc:407] Connection header received
0620 07:14:12.527455 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:12.527460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:12.527546 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:12.527654 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:12.529491 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.530705 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.532851 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.533339 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.533484 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:12.537124 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:12.537134 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:12.537142 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:12.537187 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:12.540978 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:12.541735 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:12.541745 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:12.541751 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:12.541912 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:12.542409 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:12.542417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:12.542423 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:12.542850 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:12.543032 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:12.543795 (+ 763us) server_negotiation.cc:299] Negotiation successful
0620 07:14:12.544078 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":222,"threads_started":1}
I20260620 07:14:12.545261 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:12.524748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:40833 (local address 127.6.53.65:45847)
0620 07:14:12.525482 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:12.525518 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:12.527212 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:12.527917 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:12.527929 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:12.528622 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:12.529299 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:12.529321 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.530846 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.530853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:12.532681 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:12.532697 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.533558 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.533564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:12.533759 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:12.534535 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:12.534557 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:12.536824 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:12.541150 (+ 4326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:12.541156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:12.541168 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:12.541604 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:12.542030 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:12.542033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:12.542036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:12.542233 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:12.543021 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:12.543028 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:12.543357 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 07:14:12.543596 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 07:14:12.543994 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":315,"threads_started":1}
I20260620 07:14:12.546990 24795 heartbeater.cc:347] Connected to a master server at 127.6.53.126:40833
I20260620 07:14:12.547324 24795 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:12.548274 24795 heartbeater.cc:511] Master 127.6.53.126:40833 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
I20260620 07:14:12.550668 24606 ts_manager.cc:194] Registered new tserver with Master: a13b3ab381d643828a29a4558fa7527c (127.6.53.65:44883)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
I20260620 07:14:12.552426 24606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:45847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:14:12.578579 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40833
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:12.891426 24801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:12.891722 24801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:12.891876 24801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:12.898972 24801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:12.899112 24801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:12.899273 24801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:14:12.908697 24801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:12.909870 24801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:12.911126 24801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:12.923445 24806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:12.923477 24809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:12.923532 24807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:12.924722 24801 server_base.cc:1034] running on GCE node
I20260620 07:14:12.925450 24801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:12.926298 24801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:12.927497 24801 hybrid_clock.cc:648] HybridClock initialized: now 1781939652927433 us; error 60 us; skew 500 ppm
I20260620 07:14:12.927805 24801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:12 dist-test-slave-4n6h krb5kdc[24561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:12.934945 24801 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:14:12.936326 24801 webserver.cc:460] Webserver started at http://127.6.53.66:38135/ using document root <none> and password file <none>
I20260620 07:14:12.936990 24801 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:12.937090 24801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:12.937412 24801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:12.940471 24801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "50cd34797444401da1bb6b32a085e6fa"
format_stamp: "Formatted at 2026-06-20 07:14:12 on dist-test-slave-4n6h"
server_key: "5bd7c6ee63131e59a904b7887768cbcc"
server_key_iv: "209afd3d11c4c5fcca6d19969c674c75"
server_key_version: "encryptionkey@0"
I20260620 07:14:12.941430 24801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "50cd34797444401da1bb6b32a085e6fa"
format_stamp: "Formatted at 2026-06-20 07:14:12 on dist-test-slave-4n6h"
server_key: "5bd7c6ee63131e59a904b7887768cbcc"
server_key_iv: "209afd3d11c4c5fcca6d19969c674c75"
server_key_version: "encryptionkey@0"
I20260620 07:14:12.947504 24801 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 07:14:12.951447 24816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:12.952943 24801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 07:14:12.953083 24801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "50cd34797444401da1bb6b32a085e6fa"
format_stamp: "Formatted at 2026-06-20 07:14:12 on dist-test-slave-4n6h"
server_key: "5bd7c6ee63131e59a904b7887768cbcc"
server_key_iv: "209afd3d11c4c5fcca6d19969c674c75"
server_key_version: "encryptionkey@0"
I20260620 07:14:12.953280 24801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:12.966166 24801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:12.969830 24801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:12.970055 24801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:12.971046 24801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:12.972206 24801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:12.972275 24801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:12.972353 24801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:12.972397 24801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:13.004968 24801 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:42551
I20260620 07:14:13.005136 24928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:42551 every 8 connection(s)
I20260620 07:14:13.006229 24801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:14:13.010581 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 24801
I20260620 07:14:13.010862 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:14:13.011353 6357 external_mini_cluster.cc:1428] Setting key 71fdecc449393473832e9da25d42e1e6
Jun 20 07:14:13 dist-test-slave-4n6h krb5kdc[24561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:14:13.026211 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.008818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:56109 (local address 127.6.53.126:40833)
0620 07:14:13.009033 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:13.009040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:13.010159 (+ 1119us) server_negotiation.cc:407] Connection header received
0620 07:14:13.012262 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:13.012268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:13.012361 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:13.012473 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:13.014056 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.014834 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.016337 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.016754 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.016894 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.020461 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:13.020466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:13.020469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:13.020500 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:13.023159 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:13.024035 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:13.024038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:13.024041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:13.024175 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:13.024627 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:13.024630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:13.024632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:13.024874 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:13.024997 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:13.025818 (+ 821us) server_negotiation.cc:299] Negotiation successful
0620 07:14:13.025976 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:14:13.027695 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.009330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:40833 (local address 127.6.53.66:56109)
0620 07:14:13.010018 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:13.010053 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:13.011971 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:13.012696 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:13.012704 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:13.013330 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:13.013892 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:13.013908 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.014982 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.014992 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:13.016197 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:13.016208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.016969 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.016976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:13.017193 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.017878 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:13.017898 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:13.020241 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:13.023305 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:13.023317 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:13.023335 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:13.023931 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:13.024281 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:13.024288 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:13.024294 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:13.024526 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:13.025010 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:13.025020 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:13.025441 (+ 421us) client_negotiation.cc:769] Sending connection context
0620 07:14:13.025744 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 07:14:13.026306 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":294,"threads_started":1}
I20260620 07:14:13.029179 24929 heartbeater.cc:347] Connected to a master server at 127.6.53.126:40833
I20260620 07:14:13.029507 24929 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:13.030171 24929 heartbeater.cc:511] Master 127.6.53.126:40833 requested a full tablet report, sending...
I20260620 07:14:13.031805 24606 ts_manager.cc:194] Registered new tserver with Master: 50cd34797444401da1bb6b32a085e6fa (127.6.53.66:42551)
I20260620 07:14:13.032584 24606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:56109
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:14:13.063560 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40833
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:13.382601 24934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:13.382869 24934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:13.382974 24934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:13.390618 24934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:13.390727 24934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:13.390856 24934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:14:13.400488 24934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:40833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:13.401705 24934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:13.403007 24934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:13.416280 24942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:13.416447 24939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:13.416440 24940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:13.417534 24934 server_base.cc:1034] running on GCE node
I20260620 07:14:13.418136 24934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:13.419215 24934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:13.420451 24934 hybrid_clock.cc:648] HybridClock initialized: now 1781939653420396 us; error 61 us; skew 500 ppm
I20260620 07:14:13.420781 24934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:13 dist-test-slave-4n6h krb5kdc[24561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939653, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:13.424095 24934 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:14:13.425736 24934 webserver.cc:460] Webserver started at http://127.6.53.67:41053/ using document root <none> and password file <none>
I20260620 07:14:13.426491 24934 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:13.426667 24934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:13.427021 24934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:13.429848 24934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "c081f04c1a8e465eb5208f2b139ea167"
format_stamp: "Formatted at 2026-06-20 07:14:13 on dist-test-slave-4n6h"
server_key: "586fdf2ea87264d6842f3ff695d13a4a"
server_key_iv: "a45ea1a5a75b2f13a6e516c6879b6e35"
server_key_version: "encryptionkey@0"
I20260620 07:14:13.430716 24934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "c081f04c1a8e465eb5208f2b139ea167"
format_stamp: "Formatted at 2026-06-20 07:14:13 on dist-test-slave-4n6h"
server_key: "586fdf2ea87264d6842f3ff695d13a4a"
server_key_iv: "a45ea1a5a75b2f13a6e516c6879b6e35"
server_key_version: "encryptionkey@0"
I20260620 07:14:13.435801 24934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 07:14:13.439301 24949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:13.440587 24934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 07:14:13.440769 24934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "c081f04c1a8e465eb5208f2b139ea167"
format_stamp: "Formatted at 2026-06-20 07:14:13 on dist-test-slave-4n6h"
server_key: "586fdf2ea87264d6842f3ff695d13a4a"
server_key_iv: "a45ea1a5a75b2f13a6e516c6879b6e35"
server_key_version: "encryptionkey@0"
I20260620 07:14:13.440982 24934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:13.461018 24934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:13.465164 24934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:13.465445 24934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:13.466390 24934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:13.467622 24934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:13.467720 24934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:13.467865 24934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:13.467932 24934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:13.500272 24934 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:41033
I20260620 07:14:13.500315 25061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:41033 every 8 connection(s)
I20260620 07:14:13.501539 24934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:14:13.507055 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 24934
I20260620 07:14:13.507256 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:14:13.507678 6357 external_mini_cluster.cc:1428] Setting key 7245f50482584efcae0515dcbffb1060
Jun 20 07:14:13 dist-test-slave-4n6h krb5kdc[24561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939653, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:14:13.521588 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.504288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:58937 (local address 127.6.53.126:40833)
0620 07:14:13.504556 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:13.504561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:13.505629 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 07:14:13.507546 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:13.507553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:13.507640 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:13.507753 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:13.509690 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.510680 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.512097 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.512499 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.512595 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.515674 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:13.515678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:13.515681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:13.515709 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:13.518294 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:13.519132 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:13.519135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:13.519138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:13.519258 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:13.519737 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:13.519746 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:13.519748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:13.519994 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:13.520104 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:13.520976 (+ 872us) server_negotiation.cc:299] Negotiation successful
0620 07:14:13.521233 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260620 07:14:13.522572 25064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.504807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:40833 (local address 127.6.53.67:58937)
0620 07:14:13.505476 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:13.505514 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:13.507261 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:13.508083 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:13.508095 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:13.508802 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:13.509530 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:13.509554 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.510825 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.510828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:13.511982 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:13.511993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.512668 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.512673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:13.512847 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.513437 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:13.513461 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:13.515500 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:13.518557 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:13.518565 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:13.518577 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:13.519020 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:13.519383 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:13.519388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:13.519391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:13.519606 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:13.520110 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:13.520117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:13.520522 (+ 405us) client_negotiation.cc:769] Sending connection context
0620 07:14:13.520863 (+ 341us) client_negotiation.cc:240] Negotiation successful
0620 07:14:13.521346 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":275,"threads_started":1}
I20260620 07:14:13.523882 25062 heartbeater.cc:347] Connected to a master server at 127.6.53.126:40833
I20260620 07:14:13.524183 25062 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:13.524991 25062 heartbeater.cc:511] Master 127.6.53.126:40833 requested a full tablet report, sending...
I20260620 07:14:13.526423 24606 ts_manager.cc:194] Registered new tserver with Master: c081f04c1a8e465eb5208f2b139ea167 (127.6.53.67:41033)
I20260620 07:14:13.527215 24606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:58937
I20260620 07:14:13.534528 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:14:13.551630 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.538332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35186 (local address 127.6.53.126:40833)
0620 07:14:13.538505 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:13.538513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:13.538899 (+ 386us) server_negotiation.cc:407] Connection header received
0620 07:14:13.539100 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:13.539105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:13.539163 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:13.539259 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:13.540666 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.541365 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.542820 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.543233 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.543365 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.546202 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:13.546207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:13.546217 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:13.546288 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:13.548860 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:13.549595 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:13.549603 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:13.549606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:13.549737 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:13.550176 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:13.550185 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:13.550188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:13.550528 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:13.550813 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:13.551159 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 07:14:13.551339 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:14:13.555365 24795 heartbeater.cc:503] Master 127.6.53.126:40833 was elected leader, sending a full tablet report...
I20260620 07:14:13.555483 24606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:14:13.563392 24606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:14:13.596299 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.575494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.6.53.67:41033)
0620 07:14:13.586499 (+ 11005us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:13.586512 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:13.587573 (+ 1061us) server_negotiation.cc:407] Connection header received
0620 07:14:13.587812 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:13.587821 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:13.588166 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:13.588405 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:13.589692 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.591369 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.594012 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.595028 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.595113 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.595277 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:13.595775 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 07:14:13.595935 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10763,"thread_start_us":10551,"threads_started":1}
I20260620 07:14:13.596385 25078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.574908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:41033 (local address 127.0.0.1:46378)
0620 07:14:13.587447 (+ 12539us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:13.587463 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:13.587636 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:13.588790 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:13.588801 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:13.588849 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:13.589462 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:13.589477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.591533 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.591539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:13.593831 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:13.593845 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.595436 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.595440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:13.595562 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.595592 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:14:13.595737 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 07:14:13.595918 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12398,"mutex_wait_us":3,"thread_start_us":308,"threads_started":1}
I20260620 07:14:13.598862 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.576560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:44883 (local address 127.0.0.1:35034)
0620 07:14:13.589330 (+ 12770us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:13.589358 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:13.589529 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:13.590807 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:13.590812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:13.590837 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:13.591196 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:13.591206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.593273 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.593281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:13.595713 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:13.595729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.597943 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.597954 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:13.598132 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.598174 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 07:14:13.598294 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 07:14:13.598439 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12589,"mutex_wait_us":49,"thread_start_us":10600,"threads_started":1}
I20260620 07:14:13.598898 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.574981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.6.53.66:42551)
0620 07:14:13.591512 (+ 16531us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:13.591518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:13.591538 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:14:13.591607 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:13.591612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:13.591749 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:13.591937 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:13.592493 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.593890 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.596462 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.597831 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.597963 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.598129 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:13.598415 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 07:14:13.598579 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16385,"thread_start_us":16276,"threads_started":1}
I20260620 07:14:13.598937 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.574694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:42551 (local address 127.0.0.1:59264)
0620 07:14:13.575566 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:13.575623 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:13.575775 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:13.591925 (+ 16150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:13.591930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:13.591954 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:13.592337 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:13.592347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.594051 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.594056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:13.596259 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:13.596272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.598026 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.598032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:13.598207 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.598238 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:14:13.598361 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 07:14:13.598499 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":2,"thread_start_us":14936,"threads_started":1}
I20260620 07:14:13.599675 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.576743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35034 (local address 127.6.53.65:44883)
0620 07:14:13.589800 (+ 13057us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:13.589812 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:13.589841 (+ 29us) server_negotiation.cc:407] Connection header received
0620 07:14:13.589942 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:13.589951 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:13.590156 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:13.590564 (+ 408us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:13.591407 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.592938 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.595898 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.597433 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.597561 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.597789 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:13.599239 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0620 07:14:13.599444 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12876,"thread_start_us":13538,"threads_started":1}
I20260620 07:14:13.600669 24997 tablet_service.cc:1467] Processing CreateTablet for tablet c583cf3164144e149bc007a957084807 (DEFAULT_TABLE table=test-table [id=94f5bacb3d654fa2bf942fa6ba2113a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:13.602109 24997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c583cf3164144e149bc007a957084807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:13.602313 24864 tablet_service.cc:1467] Processing CreateTablet for tablet c583cf3164144e149bc007a957084807 (DEFAULT_TABLE table=test-table [id=94f5bacb3d654fa2bf942fa6ba2113a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:13.603451 24864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c583cf3164144e149bc007a957084807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:13.605015 24730 tablet_service.cc:1467] Processing CreateTablet for tablet c583cf3164144e149bc007a957084807 (DEFAULT_TABLE table=test-table [id=94f5bacb3d654fa2bf942fa6ba2113a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:13.606309 24730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c583cf3164144e149bc007a957084807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:13.614106 25080 tablet_bootstrap.cc:492] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa: Bootstrap starting.
I20260620 07:14:13.616184 25081 tablet_bootstrap.cc:492] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167: Bootstrap starting.
I20260620 07:14:13.617646 25080 tablet_bootstrap.cc:654] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:13.618930 25080 log.cc:824] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:13.619536 25081 tablet_bootstrap.cc:654] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:13.619871 25082 tablet_bootstrap.cc:492] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c: Bootstrap starting.
I20260620 07:14:13.620779 25081 log.cc:824] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:13.622999 25080 tablet_bootstrap.cc:492] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa: No bootstrap required, opened a new log
I20260620 07:14:13.623165 25082 tablet_bootstrap.cc:654] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:13.623266 25080 ts_tablet_manager.cc:1397] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 07:14:13.623661 25081 tablet_bootstrap.cc:492] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167: No bootstrap required, opened a new log
I20260620 07:14:13.623917 25081 ts_tablet_manager.cc:1397] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 07:14:13.624341 25082 log.cc:824] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:13.627210 25082 tablet_bootstrap.cc:492] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c: No bootstrap required, opened a new log
I20260620 07:14:13.627460 25082 ts_tablet_manager.cc:1397] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260620 07:14:13.631219 25080 raft_consensus.cc:348] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cd34797444401da1bb6b32a085e6fa" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42551 } } peers { permanent_uuid: "c081f04c1a8e465eb5208f2b139ea167" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 41033 } } peers { permanent_uuid: "a13b3ab381d643828a29a4558fa7527c" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 44883 } }
I20260620 07:14:13.631592 25080 raft_consensus.cc:374] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:13.631727 25080 raft_consensus.cc:729] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50cd34797444401da1bb6b32a085e6fa, State: Initialized, Role: FOLLOWER
I20260620 07:14:13.632309 25080 consensus_queue.cc:260] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cd34797444401da1bb6b32a085e6fa" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42551 } } peers { permanent_uuid: "c081f04c1a8e465eb5208f2b139ea167" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 41033 } } peers { permanent_uuid: "a13b3ab381d643828a29a4558fa7527c" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 44883 } }
I20260620 07:14:13.633369 24929 heartbeater.cc:503] Master 127.6.53.126:40833 was elected leader, sending a full tablet report...
I20260620 07:14:13.634076 25080 ts_tablet_manager.cc:1428] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 07:14:13.634150 25081 raft_consensus.cc:348] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cd34797444401da1bb6b32a085e6fa" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42551 } } peers { permanent_uuid: "c081f04c1a8e465eb5208f2b139ea167" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 41033 } } peers { permanent_uuid: "a13b3ab381d643828a29a4558fa7527c" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 44883 } }
I20260620 07:14:13.634608 25081 raft_consensus.cc:374] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:13.634692 25081 raft_consensus.cc:729] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c081f04c1a8e465eb5208f2b139ea167, State: Initialized, Role: FOLLOWER
I20260620 07:14:13.635304 25081 consensus_queue.cc:260] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cd34797444401da1bb6b32a085e6fa" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42551 } } peers { permanent_uuid: "c081f04c1a8e465eb5208f2b139ea167" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 41033 } } peers { permanent_uuid: "a13b3ab381d643828a29a4558fa7527c" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 44883 } }
I20260620 07:14:13.636382 25062 heartbeater.cc:503] Master 127.6.53.126:40833 was elected leader, sending a full tablet report...
I20260620 07:14:13.636976 25081 ts_tablet_manager.cc:1428] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 07:14:13.638013 25082 raft_consensus.cc:348] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cd34797444401da1bb6b32a085e6fa" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42551 } } peers { permanent_uuid: "c081f04c1a8e465eb5208f2b139ea167" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 41033 } } peers { permanent_uuid: "a13b3ab381d643828a29a4558fa7527c" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 44883 } }
I20260620 07:14:13.638501 25082 raft_consensus.cc:374] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:13.638614 25082 raft_consensus.cc:729] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a13b3ab381d643828a29a4558fa7527c, State: Initialized, Role: FOLLOWER
I20260620 07:14:13.639319 25082 consensus_queue.cc:260] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cd34797444401da1bb6b32a085e6fa" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42551 } } peers { permanent_uuid: "c081f04c1a8e465eb5208f2b139ea167" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 41033 } } peers { permanent_uuid: "a13b3ab381d643828a29a4558fa7527c" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 44883 } }
I20260620 07:14:13.641101 25082 ts_tablet_manager.cc:1428] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260620 07:14:13.707233 25086 raft_consensus.cc:482] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:13.707504 25086 raft_consensus.cc:504] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cd34797444401da1bb6b32a085e6fa" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42551 } } peers { permanent_uuid: "c081f04c1a8e465eb5208f2b139ea167" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 41033 } } peers { permanent_uuid: "a13b3ab381d643828a29a4558fa7527c" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 44883 } }
I20260620 07:14:13.709090 25086 leader_election.cc:288] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c081f04c1a8e465eb5208f2b139ea167 (127.6.53.67:41033), a13b3ab381d643828a29a4558fa7527c (127.6.53.65:44883)
I20260620 07:14:13.716401 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.709559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:45421 (local address 127.6.53.65:44883)
0620 07:14:13.709807 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:13.709817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:13.710449 (+ 632us) server_negotiation.cc:407] Connection header received
0620 07:14:13.710623 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:13.710629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:13.710710 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:13.710813 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:13.711740 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.712597 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.714680 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.715433 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.715602 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.715717 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:13.716008 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 07:14:13.716142 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:14:13.716436 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.709636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:60151 (local address 127.6.53.67:41033)
0620 07:14:13.709863 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:13.709869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:13.710383 (+ 514us) server_negotiation.cc:407] Connection header received
0620 07:14:13.710623 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:13.710629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:13.710726 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:13.710813 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:13.711722 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.712674 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.714589 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.715392 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.715491 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.715623 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:13.716047 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 07:14:13.716183 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 07:14:13.716589 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.709335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:44883 (local address 127.6.53.66:45421)
0620 07:14:13.710362 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:13.710377 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:13.710471 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:13.710878 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:13.710883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:13.710916 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:13.711522 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:13.711536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.712779 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.712782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:13.714498 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:13.714533 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.715699 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.715704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:13.715837 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.715879 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 07:14:13.715950 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 07:14:13.716131 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":128,"thread_start_us":701,"threads_started":1}
I20260620 07:14:13.716647 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.709335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:41033 (local address 127.6.53.66:60151)
0620 07:14:13.710239 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:13.710285 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:13.710471 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:13.710878 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:13.710883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:13.710980 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:13.711542 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:13.711552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.712856 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.712862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:13.714437 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:13.714454 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.715640 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.715648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:13.715869 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.715893 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:14:13.715998 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 07:14:13.716141 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":90,"spinlock_wait_cycles":22272,"thread_start_us":486,"threads_started":1}
I20260620 07:14:13.717597 24750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c583cf3164144e149bc007a957084807" candidate_uuid: "50cd34797444401da1bb6b32a085e6fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a13b3ab381d643828a29a4558fa7527c" is_pre_election: true
I20260620 07:14:13.717587 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c583cf3164144e149bc007a957084807" candidate_uuid: "50cd34797444401da1bb6b32a085e6fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c081f04c1a8e465eb5208f2b139ea167" is_pre_election: true
I20260620 07:14:13.718081 25017 raft_consensus.cc:2454] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50cd34797444401da1bb6b32a085e6fa in term 0.
I20260620 07:14:13.718079 24750 raft_consensus.cc:2454] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50cd34797444401da1bb6b32a085e6fa in term 0.
I20260620 07:14:13.718755 24820 leader_election.cc:302] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 50cd34797444401da1bb6b32a085e6fa, a13b3ab381d643828a29a4558fa7527c, c081f04c1a8e465eb5208f2b139ea167; no voters:
I20260620 07:14:13.719161 25086 raft_consensus.cc:2781] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:14:13.719316 25086 raft_consensus.cc:482] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:14:13.719425 25086 raft_consensus.cc:3037] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:13.721482 25086 raft_consensus.cc:504] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cd34797444401da1bb6b32a085e6fa" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42551 } } peers { permanent_uuid: "c081f04c1a8e465eb5208f2b139ea167" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 41033 } } peers { permanent_uuid: "a13b3ab381d643828a29a4558fa7527c" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 44883 } }
I20260620 07:14:13.722020 25086 leader_election.cc:288] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [CANDIDATE]: Term 1 election: Requested vote from peers c081f04c1a8e465eb5208f2b139ea167 (127.6.53.67:41033), a13b3ab381d643828a29a4558fa7527c (127.6.53.65:44883)
I20260620 07:14:13.722483 24750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c583cf3164144e149bc007a957084807" candidate_uuid: "50cd34797444401da1bb6b32a085e6fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a13b3ab381d643828a29a4558fa7527c"
I20260620 07:14:13.722483 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c583cf3164144e149bc007a957084807" candidate_uuid: "50cd34797444401da1bb6b32a085e6fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c081f04c1a8e465eb5208f2b139ea167"
I20260620 07:14:13.722695 25017 raft_consensus.cc:3037] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:13.722685 24750 raft_consensus.cc:3037] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:13.725847 24750 raft_consensus.cc:2454] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50cd34797444401da1bb6b32a085e6fa in term 1.
I20260620 07:14:13.725847 25017 raft_consensus.cc:2454] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50cd34797444401da1bb6b32a085e6fa in term 1.
I20260620 07:14:13.726284 24820 leader_election.cc:302] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 50cd34797444401da1bb6b32a085e6fa, a13b3ab381d643828a29a4558fa7527c, c081f04c1a8e465eb5208f2b139ea167; no voters:
I20260620 07:14:13.726542 25086 raft_consensus.cc:2781] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:13.726985 25086 raft_consensus.cc:686] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [term 1 LEADER]: Becoming Leader. State: Replica: 50cd34797444401da1bb6b32a085e6fa, State: Running, Role: LEADER
I20260620 07:14:13.727499 25086 consensus_queue.cc:237] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cd34797444401da1bb6b32a085e6fa" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42551 } } peers { permanent_uuid: "c081f04c1a8e465eb5208f2b139ea167" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 41033 } } peers { permanent_uuid: "a13b3ab381d643828a29a4558fa7527c" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 44883 } }
I20260620 07:14:13.731788 24605 catalog_manager.cc:5495] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 50cd34797444401da1bb6b32a085e6fa (127.6.53.66). New cstate: current_term: 1 leader_uuid: "50cd34797444401da1bb6b32a085e6fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50cd34797444401da1bb6b32a085e6fa" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 42551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c081f04c1a8e465eb5208f2b139ea167" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 41033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a13b3ab381d643828a29a4558fa7527c" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 44883 } health_report { overall_health: UNKNOWN } } }
W20260620 07:14:13.753108 25063 tablet.cc:2369] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:14:13.754441 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.748844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.6.53.66:42551)
0620 07:14:13.749071 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:13.749077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:13.749097 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:14:13.749271 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:13.749276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:13.749348 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:13.749549 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:14:13.749934 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.750733 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.752135 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.752526 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.752671 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.753248 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:14:13.753762 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:14:13.753945 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:13.754106 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 07:14:13.754151 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 07:14:13.758203 24930 tablet.cc:2369] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:14:13.762353 24750 raft_consensus.cc:1261] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c [term 1 FOLLOWER]: Refusing update from remote peer 50cd34797444401da1bb6b32a085e6fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:13.762356 25017 raft_consensus.cc:1261] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167 [term 1 FOLLOWER]: Refusing update from remote peer 50cd34797444401da1bb6b32a085e6fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:13.763273 25086 consensus_queue.cc:1035] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c081f04c1a8e465eb5208f2b139ea167" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 41033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:13.763564 25091 consensus_queue.cc:1035] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a13b3ab381d643828a29a4558fa7527c" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 44883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:14:13.773856 24796 tablet.cc:2369] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:14:13.774657 25094 mvcc.cc:205] Tried to move back new op lower bound from 7298824821800902656 to 7298824821671718912. Current Snapshot: MvccSnapshot[applied={T|T < 7298824821800902656}]
I20260620 07:14:13.792109 24606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:13.792389 24606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:13.795607 24606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aadb8d787ad84e7d94ba81161534267e: Sending DeleteTablet for 3 replicas of tablet c583cf3164144e149bc007a957084807
I20260620 07:14:13.796710 24997 tablet_service.cc:1514] Processing DeleteTablet for tablet c583cf3164144e149bc007a957084807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:13 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:46378
I20260620 07:14:13.796967 24864 tablet_service.cc:1514] Processing DeleteTablet for tablet c583cf3164144e149bc007a957084807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:13 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:59264
I20260620 07:14:13.796985 24730 tablet_service.cc:1514] Processing DeleteTablet for tablet c583cf3164144e149bc007a957084807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:13 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:35034
I20260620 07:14:13.801302 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 24667
I20260620 07:14:13.807869 25105 tablet_replica.cc:330] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa: stopping tablet replica
I20260620 07:14:13.808553 25105 raft_consensus.cc:2229] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:14:13.809406 25105 raft_consensus.cc:2258] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:13.811841 25105 ts_tablet_manager.cc:1905] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:13.818836 25106 tablet_replica.cc:330] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c: stopping tablet replica
I20260620 07:14:13.819363 25106 raft_consensus.cc:2229] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:13.819965 25106 raft_consensus.cc:2258] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:13.820721 25104 tablet_replica.cc:330] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167: stopping tablet replica
I20260620 07:14:13.821115 25104 raft_consensus.cc:2229] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:13.821586 25104 raft_consensus.cc:2258] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:13.822597 25106 ts_tablet_manager.cc:1905] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:13.823724 25104 ts_tablet_manager.cc:1905] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:13.829870 25106 ts_tablet_manager.cc:1918] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:13.830090 25106 log.cc:1196] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/c583cf3164144e149bc007a957084807
I20260620 07:14:13.830499 25104 ts_tablet_manager.cc:1918] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:13.830678 25104 log.cc:1196] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/c583cf3164144e149bc007a957084807
I20260620 07:14:13.830756 25106 ts_tablet_manager.cc:1939] T c583cf3164144e149bc007a957084807 P a13b3ab381d643828a29a4558fa7527c: Deleting consensus metadata
I20260620 07:14:13.831187 25104 ts_tablet_manager.cc:1939] T c583cf3164144e149bc007a957084807 P c081f04c1a8e465eb5208f2b139ea167: Deleting consensus metadata
Jun 20 07:14:13 dist-test-slave-4n6h krb5kdc[24561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:14:13.832662 24593 catalog_manager.cc:4841] TS a13b3ab381d643828a29a4558fa7527c (127.6.53.65:44883): tablet c583cf3164144e149bc007a957084807 (table test-table [id=94f5bacb3d654fa2bf942fa6ba2113a0]) successfully deleted
I20260620 07:14:13.832778 24592 catalog_manager.cc:4841] TS c081f04c1a8e465eb5208f2b139ea167 (127.6.53.67:41033): tablet c583cf3164144e149bc007a957084807 (table test-table [id=94f5bacb3d654fa2bf942fa6ba2113a0]) successfully deleted
I20260620 07:14:13.833635 25105 ts_tablet_manager.cc:1918] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:13.833812 25105 log.cc:1196] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/c583cf3164144e149bc007a957084807
I20260620 07:14:13.834412 25105 ts_tablet_manager.cc:1939] T c583cf3164144e149bc007a957084807 P 50cd34797444401da1bb6b32a085e6fa: Deleting consensus metadata
I20260620 07:14:13.836141 24590 catalog_manager.cc:4841] TS 50cd34797444401da1bb6b32a085e6fa (127.6.53.66:42551): tablet c583cf3164144e149bc007a957084807 (table test-table [id=94f5bacb3d654fa2bf942fa6ba2113a0]) successfully deleted
I20260620 07:14:13.844446 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.802557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.6.53.65:44883)
0620 07:14:13.802780 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:13.802789 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:13.820997 (+ 18208us) server_negotiation.cc:407] Connection header received
0620 07:14:13.821576 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:13.821584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:13.821682 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:13.821761 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:13.823735 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.824898 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.826918 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.827608 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.827706 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.836817 (+ 9111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:13.836832 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:13.836849 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:13.836897 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:13.840819 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:13.841685 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:13.841695 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:13.841699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:13.841857 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:13.842411 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:13.842416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:13.842419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:13.842944 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:13.843252 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:13.843773 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 07:14:13.844028 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:14:13.845201 24790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:13.963188 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 24667
I20260620 07:14:13.988427 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 24801
Jun 20 07:14:13 dist-test-slave-4n6h krb5kdc[24561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:14:14.006234 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:13.989220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59284 (local address 127.6.53.66:42551)
0620 07:14:13.989422 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:13.989427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:13.989450 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:14:13.989554 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:13.989560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:13.989637 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:13.989755 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:13.991026 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.992229 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.993440 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:13.994133 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:13.994288 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:13.998699 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:13.998718 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:13.998744 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:13.998788 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:14.003068 (+ 4280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:14.003739 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:14.003748 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:14.003753 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:14.003967 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:14.004477 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:14.004483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:14.004487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:14.004951 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:14.005317 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:14.005605 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 07:14:14.005836 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:14:14.007066 24924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:14.124228 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 24801
I20260620 07:14:14.142486 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 24934
Jun 20 07:14:14 dist-test-slave-4n6h krb5kdc[24561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:14:14.156677 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:14.143256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.6.53.67:41033)
0620 07:14:14.143427 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:14.143446 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:14.143473 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:14:14.143576 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:14.143580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:14.143650 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:14.143740 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:14.145075 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:14.145919 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:14.147271 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:14.147645 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:14.147742 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:14.151912 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:14.151926 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:14.151941 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:14.151979 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:14.154548 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:14.155116 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:14.155122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:14.155125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:14.155260 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:14.155547 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:14.155550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:14.155551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:14.155853 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:14.156061 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:14.156282 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 07:14:14.156432 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:14:14.157277 25057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:14.245268 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 24934
I20260620 07:14:14.261670 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 24574
I20260620 07:14:14.262827 24636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:14.369566 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 24574
2026-06-20T07:14:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4182 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T07:14:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:14Z Disabled control of system clock
I20260620 07:14:14.442531 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:38049
--webserver_interface=127.6.53.126
--webserver_port=0
--builtin_ntp_servers=127.6.53.84:46601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:38049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:14:14.761957 25115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:14.762238 25115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:14.762393 25115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:14.769388 25115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:14.769487 25115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:14.769533 25115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:14.769591 25115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:14.769634 25115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:14.778513 25115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:46601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:38049
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:38049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:14.779659 25115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:14.781303 25115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:14.792428 25123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:14.792546 25120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:14.792588 25121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:14.793080 25115 server_base.cc:1034] running on GCE node
I20260620 07:14:14.793803 25115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:14.795136 25115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:14.796355 25115 hybrid_clock.cc:648] HybridClock initialized: now 1781939654796324 us; error 43 us; skew 500 ppm
I20260620 07:14:14.796690 25115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:14:14.798887 25115 webserver.cc:460] Webserver started at http://127.6.53.126:42311/ using document root <none> and password file <none>
I20260620 07:14:14.799526 25115 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:14.799652 25115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:14.799952 25115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:14.802734 25115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "33210d7457d547f39c82cba10251c3d2"
format_stamp: "Formatted at 2026-06-20 07:14:14 on dist-test-slave-4n6h"
server_key: "c457ee6551243a10c04d509bf16ef864"
server_key_iv: "ed7a8f82c0a3f089cd467c707600a721"
server_key_version: "encryptionkey@0"
I20260620 07:14:14.803503 25115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "33210d7457d547f39c82cba10251c3d2"
format_stamp: "Formatted at 2026-06-20 07:14:14 on dist-test-slave-4n6h"
server_key: "c457ee6551243a10c04d509bf16ef864"
server_key_iv: "ed7a8f82c0a3f089cd467c707600a721"
server_key_version: "encryptionkey@0"
I20260620 07:14:14.808339 25115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:14:14.811769 25129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:14.812912 25115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 07:14:14.813093 25115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "33210d7457d547f39c82cba10251c3d2"
format_stamp: "Formatted at 2026-06-20 07:14:14 on dist-test-slave-4n6h"
server_key: "c457ee6551243a10c04d509bf16ef864"
server_key_iv: "ed7a8f82c0a3f089cd467c707600a721"
server_key_version: "encryptionkey@0"
I20260620 07:14:14.813295 25115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:14.840543 25115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:14.841419 25115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:14.841737 25115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:14.860178 25115 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:38049
I20260620 07:14:14.860198 25180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:38049 every 8 connection(s)
I20260620 07:14:14.861567 25115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:14.862792 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 25115
I20260620 07:14:14.862963 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:14.863508 6357 external_mini_cluster.cc:1428] Setting key ee7dc44f7b0e103aea677ab1db44d24e
I20260620 07:14:14.866046 25181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:14.877424 25184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:14.865385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.6.53.126:38049)
0620 07:14:14.866508 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:14.866518 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:14.866561 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:14:14.868262 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:14.868283 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:14.868742 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:14.869070 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:14.869666 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:14.871145 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:14.873057 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:14.873725 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:14.873862 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:14.875114 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:14.875122 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:14.875135 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:14.875754 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:14.875842 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:14.875873 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:14.875971 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:14.876193 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 07:14:14.876532 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":381,"threads_started":1}
I20260620 07:14:14.878216 25181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2: Bootstrap starting.
I20260620 07:14:14.881309 25181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:14.882416 25181 log.cc:824] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:14.884887 25181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2: No bootstrap required, opened a new log
I20260620 07:14:14.892091 25181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33210d7457d547f39c82cba10251c3d2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38049 } }
I20260620 07:14:14.892387 25181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:14.892455 25181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33210d7457d547f39c82cba10251c3d2, State: Initialized, Role: FOLLOWER
I20260620 07:14:14.892925 25181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33210d7457d547f39c82cba10251c3d2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38049 } }
I20260620 07:14:14.893118 25181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:14.893213 25181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:14.893333 25181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:14.895637 25181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33210d7457d547f39c82cba10251c3d2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38049 } }
I20260620 07:14:14.896066 25181 leader_election.cc:302] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33210d7457d547f39c82cba10251c3d2; no voters:
I20260620 07:14:14.896575 25181 leader_election.cc:288] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:14.896718 25186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:14.897455 25186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 33210d7457d547f39c82cba10251c3d2, State: Running, Role: LEADER
I20260620 07:14:14.897921 25186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33210d7457d547f39c82cba10251c3d2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38049 } }
I20260620 07:14:14.898623 25181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:14.900806 25187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33210d7457d547f39c82cba10251c3d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33210d7457d547f39c82cba10251c3d2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38049 } } }
I20260620 07:14:14.900806 25188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33210d7457d547f39c82cba10251c3d2. Latest consensus state: current_term: 1 leader_uuid: "33210d7457d547f39c82cba10251c3d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33210d7457d547f39c82cba10251c3d2" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 38049 } } }
I20260620 07:14:14.901273 25187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:14.901268 25188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:14.903904 25191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:14.909168 25191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:14.918035 25191 catalog_manager.cc:1347] Generated new cluster ID: acfcb03669d541ae8ec16b714f4f8f37
I20260620 07:14:14.918143 25191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:14.950457 25191 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:14.951726 25191 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:14.968109 25191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2: Generated new TSK 0
I20260620 07:14:14.968873 25191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:14.973477 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38049
--builtin_ntp_servers=127.6.53.84:46601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:14:15.333300 25205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:15.333545 25205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:15.333637 25205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:15.340656 25205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:15.340744 25205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:15.340845 25205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:14:15.349675 25205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:46601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:15.350838 25205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:15.352473 25205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:15.365139 25210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:15.365245 25211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:15.365805 25213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:15.365931 25205 server_base.cc:1034] running on GCE node
I20260620 07:14:15.366559 25205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:15.367434 25205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:15.368635 25205 hybrid_clock.cc:648] HybridClock initialized: now 1781939655368576 us; error 53 us; skew 500 ppm
I20260620 07:14:15.368953 25205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:14:15.371168 25205 webserver.cc:460] Webserver started at http://127.6.53.65:40321/ using document root <none> and password file <none>
I20260620 07:14:15.371836 25205 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:15.371948 25205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:15.372262 25205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:15.375012 25205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "78b5a2db0bcd41b5a6cb5a29e820ba26"
format_stamp: "Formatted at 2026-06-20 07:14:15 on dist-test-slave-4n6h"
server_key: "a7bd174ff8afb08677dd1ca824ca77a3"
server_key_iv: "264cd7df8b73737df26550db9a1dc4e5"
server_key_version: "encryptionkey@0"
I20260620 07:14:15.375783 25205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "78b5a2db0bcd41b5a6cb5a29e820ba26"
format_stamp: "Formatted at 2026-06-20 07:14:15 on dist-test-slave-4n6h"
server_key: "a7bd174ff8afb08677dd1ca824ca77a3"
server_key_iv: "264cd7df8b73737df26550db9a1dc4e5"
server_key_version: "encryptionkey@0"
I20260620 07:14:15.380643 25205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:14:15.384122 25219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:15.385416 25205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:14:15.385565 25205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "78b5a2db0bcd41b5a6cb5a29e820ba26"
format_stamp: "Formatted at 2026-06-20 07:14:15 on dist-test-slave-4n6h"
server_key: "a7bd174ff8afb08677dd1ca824ca77a3"
server_key_iv: "264cd7df8b73737df26550db9a1dc4e5"
server_key_version: "encryptionkey@0"
I20260620 07:14:15.385778 25205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:15.401481 25205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:15.402292 25205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:15.402559 25205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:15.403470 25205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:15.404991 25205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:15.405123 25205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:15.405241 25205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:15.405323 25205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:15.437304 25205 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:34529
I20260620 07:14:15.437386 25331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:34529 every 8 connection(s)
I20260620 07:14:15.438702 25205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:14:15.441392 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 25205
I20260620 07:14:15.441591 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:14:15.441990 6357 external_mini_cluster.cc:1428] Setting key 8d973d65d2859aac5df736820ee05d89
I20260620 07:14:15.445147 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38049
--builtin_ntp_servers=127.6.53.84:46601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:14:15.452941 25334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.441236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:40077 (local address 127.6.53.126:38049)
0620 07:14:15.441757 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:15.441763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:15.442934 (+ 1171us) server_negotiation.cc:407] Connection header received
0620 07:14:15.445336 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:15.445344 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:15.445439 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:15.445532 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:15.446948 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.447648 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.450034 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.450504 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.450635 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.452191 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:15.452196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:15.452200 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:15.452218 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:15.452255 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:15.452267 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:15.452335 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:15.452601 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 07:14:15.452680 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":251,"threads_started":1}
I20260620 07:14:15.454195 25335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.441779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:38049 (local address 127.6.53.65:40077)
0620 07:14:15.442786 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.442827 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:15.445018 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:15.445799 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:15.445814 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:15.445933 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:15.446780 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.446808 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.447799 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.447807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.449884 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.449901 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.450743 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.450753 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.451074 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.451845 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:14:15.451850 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:15.451902 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:14:15.452001 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:15.452359 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:15.452372 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:15.452382 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 07:14:15.452573 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 07:14:15.452996 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":301,"threads_started":1}
I20260620 07:14:15.455853 25332 heartbeater.cc:347] Connected to a master server at 127.6.53.126:38049
I20260620 07:14:15.456177 25332 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:15.456893 25332 heartbeater.cc:511] Master 127.6.53.126:38049 requested a full tablet report, sending...
I20260620 07:14:15.458879 25146 ts_manager.cc:194] Registered new tserver with Master: 78b5a2db0bcd41b5a6cb5a29e820ba26 (127.6.53.65:34529)
I20260620 07:14:15.460299 25146 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.53.65:40077
W20260620 07:14:15.758992 25336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:15.759352 25336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:15.759503 25336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:15.766569 25336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:15.766718 25336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:15.766893 25336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:14:15.775679 25336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:46601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:15.776933 25336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:15.778632 25336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:15.791054 25341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:15.791014 25342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:15.791153 25344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:15.792001 25336 server_base.cc:1034] running on GCE node
I20260620 07:14:15.792640 25336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:15.793560 25336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:15.794777 25336 hybrid_clock.cc:648] HybridClock initialized: now 1781939655794743 us; error 48 us; skew 500 ppm
I20260620 07:14:15.795082 25336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:14:15.797128 25336 webserver.cc:460] Webserver started at http://127.6.53.66:40599/ using document root <none> and password file <none>
I20260620 07:14:15.797778 25336 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:15.797900 25336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:15.798219 25336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:15.800925 25336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "3b04da6f74d64797b16e65eab64885be"
format_stamp: "Formatted at 2026-06-20 07:14:15 on dist-test-slave-4n6h"
server_key: "481ae9e1b9df85e6862a82a2f8844d96"
server_key_iv: "68817ed5733954214f6188c4b6621b0b"
server_key_version: "encryptionkey@0"
I20260620 07:14:15.801690 25336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "3b04da6f74d64797b16e65eab64885be"
format_stamp: "Formatted at 2026-06-20 07:14:15 on dist-test-slave-4n6h"
server_key: "481ae9e1b9df85e6862a82a2f8844d96"
server_key_iv: "68817ed5733954214f6188c4b6621b0b"
server_key_version: "encryptionkey@0"
I20260620 07:14:15.806526 25336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260620 07:14:15.809773 25350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:15.811112 25336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:15.811273 25336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "3b04da6f74d64797b16e65eab64885be"
format_stamp: "Formatted at 2026-06-20 07:14:15 on dist-test-slave-4n6h"
server_key: "481ae9e1b9df85e6862a82a2f8844d96"
server_key_iv: "68817ed5733954214f6188c4b6621b0b"
server_key_version: "encryptionkey@0"
I20260620 07:14:15.811479 25336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:15.823243 25336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:15.823956 25336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:15.824219 25336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:15.825089 25336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:15.826509 25336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:15.826625 25336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:15.826763 25336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:15.826850 25336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:15.865098 25336 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:44245
I20260620 07:14:15.865121 25462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:44245 every 8 connection(s)
I20260620 07:14:15.866415 25336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:14:15.870434 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 25336
I20260620 07:14:15.870608 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:14:15.871155 6357 external_mini_cluster.cc:1428] Setting key 6230c3cb93f5afccac00a888d2ae67bc
I20260620 07:14:15.874181 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38049
--builtin_ntp_servers=127.6.53.84:46601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:14:15.878851 25334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.868865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:46983 (local address 127.6.53.126:38049)
0620 07:14:15.869081 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:15.869087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:15.870569 (+ 1482us) server_negotiation.cc:407] Connection header received
0620 07:14:15.872573 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:15.872579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:15.872655 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:15.872757 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:15.873852 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.874738 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.876214 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.876610 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.876736 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.878024 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:15.878031 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:15.878035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:15.878064 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:15.878093 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:15.878105 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:15.878193 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:15.878449 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 07:14:15.878543 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:14:15.879755 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.869390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:38049 (local address 127.6.53.66:46983)
0620 07:14:15.870404 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.870451 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:15.872276 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:15.873005 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:15.873016 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:15.873132 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:15.873691 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.873708 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.874886 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.874891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.876072 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.876083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.876826 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.876833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.877048 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.877721 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:14:15.877723 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:15.877764 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:14:15.877834 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:15.878205 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:15.878217 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:15.878229 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 07:14:15.878444 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 07:14:15.878808 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":339,"threads_started":1}
I20260620 07:14:15.881026 25463 heartbeater.cc:347] Connected to a master server at 127.6.53.126:38049
I20260620 07:14:15.881319 25463 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:15.881990 25463 heartbeater.cc:511] Master 127.6.53.126:38049 requested a full tablet report, sending...
I20260620 07:14:15.883422 25146 ts_manager.cc:194] Registered new tserver with Master: 3b04da6f74d64797b16e65eab64885be (127.6.53.66:44245)
I20260620 07:14:15.884197 25146 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.53.66:46983
W20260620 07:14:16.193042 25466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:16.193324 25466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:16.193501 25466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:16.200830 25466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:16.200955 25466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:16.201107 25466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:14:16.210012 25466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:46601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:38049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:16.211263 25466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:16.212896 25466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:16.225595 25472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:16.225623 25474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:16.225701 25471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:16.226365 25466 server_base.cc:1034] running on GCE node
I20260620 07:14:16.226955 25466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:16.227883 25466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:16.229120 25466 hybrid_clock.cc:648] HybridClock initialized: now 1781939656229063 us; error 48 us; skew 500 ppm
I20260620 07:14:16.229449 25466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:14:16.231695 25466 webserver.cc:460] Webserver started at http://127.6.53.67:42711/ using document root <none> and password file <none>
I20260620 07:14:16.232365 25466 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:16.232502 25466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:16.232813 25466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:16.235617 25466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "37ab839d121943169982aab4e19b127c"
format_stamp: "Formatted at 2026-06-20 07:14:16 on dist-test-slave-4n6h"
server_key: "e53bea20843d7ef61a2a1fb144c77c45"
server_key_iv: "83fadb64103bbb808aca7c945dd6d42c"
server_key_version: "encryptionkey@0"
I20260620 07:14:16.236390 25466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "37ab839d121943169982aab4e19b127c"
format_stamp: "Formatted at 2026-06-20 07:14:16 on dist-test-slave-4n6h"
server_key: "e53bea20843d7ef61a2a1fb144c77c45"
server_key_iv: "83fadb64103bbb808aca7c945dd6d42c"
server_key_version: "encryptionkey@0"
I20260620 07:14:16.241300 25466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:14:16.244849 25480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:16.246104 25466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 07:14:16.246311 25466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "37ab839d121943169982aab4e19b127c"
format_stamp: "Formatted at 2026-06-20 07:14:16 on dist-test-slave-4n6h"
server_key: "e53bea20843d7ef61a2a1fb144c77c45"
server_key_iv: "83fadb64103bbb808aca7c945dd6d42c"
server_key_version: "encryptionkey@0"
I20260620 07:14:16.246527 25466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:16.275593 25466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:16.276517 25466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:16.276803 25466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:16.277822 25466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:16.279346 25466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:16.279464 25466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:16.279595 25466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:16.279686 25466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:16.312848 25466 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:34745
I20260620 07:14:16.312930 25592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:34745 every 8 connection(s)
I20260620 07:14:16.314080 25466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:14:16.318720 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 25466
I20260620 07:14:16.318904 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:14:16.319303 6357 external_mini_cluster.cc:1428] Setting key cf11c00aae1754dc3000359b6eed566f
I20260620 07:14:16.326103 25334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.316743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:57439 (local address 127.6.53.126:38049)
0620 07:14:16.316924 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:16.316928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:16.318215 (+ 1287us) server_negotiation.cc:407] Connection header received
0620 07:14:16.320187 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:16.320193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:16.320277 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:16.320380 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:16.321326 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.322073 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.323576 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.323964 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.324094 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.325314 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:16.325319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:16.325322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:16.325338 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:16.325366 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:16.325377 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:16.325466 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:16.325676 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 07:14:16.325766 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:14:16.326963 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.317149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:38049 (local address 127.6.53.67:57439)
0620 07:14:16.318065 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:16.318102 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:16.319911 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:16.320584 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:16.320592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:16.320667 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:16.321174 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:16.321193 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.322221 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.322225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:16.323431 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:16.323442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.324169 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.324175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:16.324383 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.325027 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:14:16.325030 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:16.325070 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:14:16.325137 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:16.325466 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:16.325478 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:16.325487 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 07:14:16.325665 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 07:14:16.326010 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":266,"threads_started":1}
I20260620 07:14:16.328191 25593 heartbeater.cc:347] Connected to a master server at 127.6.53.126:38049
I20260620 07:14:16.328420 25593 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:16.329049 25593 heartbeater.cc:511] Master 127.6.53.126:38049 requested a full tablet report, sending...
I20260620 07:14:16.330221 25146 ts_manager.cc:194] Registered new tserver with Master: 37ab839d121943169982aab4e19b127c (127.6.53.67:34745)
I20260620 07:14:16.331066 25146 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.53.67:57439
I20260620 07:14:16.333675 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:14:16.343334 25334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.337730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.6.53.126:38049)
0620 07:14:16.337886 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:16.337890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:16.338363 (+ 473us) server_negotiation.cc:407] Connection header received
0620 07:14:16.338488 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:16.338492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:16.338559 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:16.338642 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:16.339095 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.339757 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.341148 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.341569 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.341668 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.342722 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:16.342727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:16.342730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:16.342746 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:16.342774 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:16.342785 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:16.342888 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:16.343091 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 07:14:16.343131 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:14:16.347085 25146 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:14:16.353829 25146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:14:16.382335 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.368615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.6.53.65:34529)
0620 07:14:16.370307 (+ 1692us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:16.370322 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:16.371167 (+ 845us) server_negotiation.cc:407] Connection header received
0620 07:14:16.371307 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:16.371319 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:16.371629 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:16.372332 (+ 703us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:16.373723 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.375970 (+ 2247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.378122 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.381053 (+ 2931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.381235 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.381486 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:16.381794 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 07:14:16.381960 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1356,"thread_start_us":1215,"threads_started":1}
I20260620 07:14:16.382325 25609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.367906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:34529 (local address 127.0.0.1:58920)
0620 07:14:16.370830 (+ 2924us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:16.370853 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:16.370992 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:16.372930 (+ 1938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:16.372942 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:16.372988 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:16.373456 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:16.373473 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.376166 (+ 2693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.376171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:16.377766 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:16.377780 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.381275 (+ 3495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.381282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:16.381560 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.381601 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 07:14:16.381746 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 07:14:16.381945 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2779,"thread_start_us":2591,"threads_started":1}
I20260620 07:14:16.384214 25605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.365990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:44245 (local address 127.0.0.1:45062)
0620 07:14:16.367812 (+ 1822us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:16.367871 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:16.368065 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:16.375022 (+ 6957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:16.375030 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:16.375066 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:16.375622 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:16.375638 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.378869 (+ 3231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.378878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:16.381407 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:16.381426 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.383376 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.383382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:16.383547 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.383578 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:14:16.383682 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 07:14:16.383812 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1493,"mutex_wait_us":5,"thread_start_us":1275,"threads_started":1}
I20260620 07:14:16.384235 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.366635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.6.53.66:44245)
0620 07:14:16.374184 (+ 7549us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:16.374197 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:16.374233 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:14:16.374389 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:16.374400 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:16.374742 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:16.375867 (+ 1125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:16.376209 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.378672 (+ 2463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.381646 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.383148 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.383287 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.383516 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:16.383736 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 07:14:16.383898 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7318,"thread_start_us":7144,"threads_started":1}
I20260620 07:14:16.386986 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.366090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:34745 (local address 127.0.0.1:56122)
0620 07:14:16.368719 (+ 2629us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:16.368741 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:16.368865 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:16.379518 (+ 10653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:16.379525 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:16.379552 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:16.380131 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:16.380147 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.382213 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.382219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:16.384356 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:16.384372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.386167 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.386176 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:16.386380 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.386413 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:14:16.386504 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 07:14:16.386626 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2464,"mutex_wait_us":1,"thread_start_us":2925,"threads_started":1}
I20260620 07:14:16.387401 25607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.366450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.6.53.67:34745)
0620 07:14:16.378879 (+ 12429us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:16.378887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:16.378923 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:14:16.379010 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:16.379017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:16.379240 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:16.379494 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:16.380368 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.382059 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.384575 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.385948 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.386503 (+ 555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.386761 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:16.386827 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 07:14:16.387000 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12253,"thread_start_us":12066,"threads_started":1}
I20260620 07:14:16.387099 25267 tablet_service.cc:1467] Processing CreateTablet for tablet 39041d0e54b9461694a69a41d5f1af57 (DEFAULT_TABLE table=test-table [id=5f6fa17003a4435195b4dca96be2e7df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:16.388607 25267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39041d0e54b9461694a69a41d5f1af57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:16.388213 25398 tablet_service.cc:1467] Processing CreateTablet for tablet 39041d0e54b9461694a69a41d5f1af57 (DEFAULT_TABLE table=test-table [id=5f6fa17003a4435195b4dca96be2e7df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:16.389500 25398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39041d0e54b9461694a69a41d5f1af57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:16.390448 25528 tablet_service.cc:1467] Processing CreateTablet for tablet 39041d0e54b9461694a69a41d5f1af57 (DEFAULT_TABLE table=test-table [id=5f6fa17003a4435195b4dca96be2e7df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:16.391558 25528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39041d0e54b9461694a69a41d5f1af57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:16.403262 25611 tablet_bootstrap.cc:492] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c: Bootstrap starting.
I20260620 07:14:16.406347 25612 tablet_bootstrap.cc:492] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26: Bootstrap starting.
I20260620 07:14:16.406718 25611 tablet_bootstrap.cc:654] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:16.407837 25613 tablet_bootstrap.cc:492] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be: Bootstrap starting.
I20260620 07:14:16.408136 25611 log.cc:824] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:16.410157 25612 tablet_bootstrap.cc:654] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:16.411566 25613 tablet_bootstrap.cc:654] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:16.411621 25611 tablet_bootstrap.cc:492] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c: No bootstrap required, opened a new log
I20260620 07:14:16.411607 25612 log.cc:824] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:16.411902 25611 ts_tablet_manager.cc:1397] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 07:14:16.412680 25613 log.cc:824] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:16.414759 25612 tablet_bootstrap.cc:492] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26: No bootstrap required, opened a new log
I20260620 07:14:16.415050 25612 ts_tablet_manager.cc:1397] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 07:14:16.415279 25613 tablet_bootstrap.cc:492] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be: No bootstrap required, opened a new log
I20260620 07:14:16.415512 25613 ts_tablet_manager.cc:1397] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 07:14:16.423405 25613 raft_consensus.cc:348] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b04da6f74d64797b16e65eab64885be" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44245 } } peers { permanent_uuid: "37ab839d121943169982aab4e19b127c" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34745 } } peers { permanent_uuid: "78b5a2db0bcd41b5a6cb5a29e820ba26" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 34529 } }
I20260620 07:14:16.423342 25611 raft_consensus.cc:348] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b04da6f74d64797b16e65eab64885be" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44245 } } peers { permanent_uuid: "37ab839d121943169982aab4e19b127c" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34745 } } peers { permanent_uuid: "78b5a2db0bcd41b5a6cb5a29e820ba26" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 34529 } }
I20260620 07:14:16.423770 25613 raft_consensus.cc:374] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:16.423822 25611 raft_consensus.cc:374] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:16.423853 25613 raft_consensus.cc:729] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b04da6f74d64797b16e65eab64885be, State: Initialized, Role: FOLLOWER
I20260620 07:14:16.423928 25611 raft_consensus.cc:729] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ab839d121943169982aab4e19b127c, State: Initialized, Role: FOLLOWER
I20260620 07:14:16.424450 25613 consensus_queue.cc:260] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b04da6f74d64797b16e65eab64885be" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44245 } } peers { permanent_uuid: "37ab839d121943169982aab4e19b127c" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34745 } } peers { permanent_uuid: "78b5a2db0bcd41b5a6cb5a29e820ba26" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 34529 } }
I20260620 07:14:16.424656 25611 consensus_queue.cc:260] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b04da6f74d64797b16e65eab64885be" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44245 } } peers { permanent_uuid: "37ab839d121943169982aab4e19b127c" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34745 } } peers { permanent_uuid: "78b5a2db0bcd41b5a6cb5a29e820ba26" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 34529 } }
I20260620 07:14:16.425631 25463 heartbeater.cc:503] Master 127.6.53.126:38049 was elected leader, sending a full tablet report...
I20260620 07:14:16.426126 25613 ts_tablet_manager.cc:1428] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260620 07:14:16.426270 25612 raft_consensus.cc:348] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b04da6f74d64797b16e65eab64885be" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44245 } } peers { permanent_uuid: "37ab839d121943169982aab4e19b127c" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34745 } } peers { permanent_uuid: "78b5a2db0bcd41b5a6cb5a29e820ba26" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 34529 } }
I20260620 07:14:16.426754 25612 raft_consensus.cc:374] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:16.426880 25612 raft_consensus.cc:729] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b5a2db0bcd41b5a6cb5a29e820ba26, State: Initialized, Role: FOLLOWER
I20260620 07:14:16.427738 25593 heartbeater.cc:503] Master 127.6.53.126:38049 was elected leader, sending a full tablet report...
I20260620 07:14:16.427857 25612 consensus_queue.cc:260] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b04da6f74d64797b16e65eab64885be" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44245 } } peers { permanent_uuid: "37ab839d121943169982aab4e19b127c" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34745 } } peers { permanent_uuid: "78b5a2db0bcd41b5a6cb5a29e820ba26" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 34529 } }
I20260620 07:14:16.428298 25611 ts_tablet_manager.cc:1428] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260620 07:14:16.429255 25332 heartbeater.cc:503] Master 127.6.53.126:38049 was elected leader, sending a full tablet report...
I20260620 07:14:16.429927 25612 ts_tablet_manager.cc:1428] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260620 07:14:16.440939 25333 tablet.cc:2369] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:14:16.542196 25618 raft_consensus.cc:482] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:16.542497 25618 raft_consensus.cc:504] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b04da6f74d64797b16e65eab64885be" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44245 } } peers { permanent_uuid: "37ab839d121943169982aab4e19b127c" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34745 } } peers { permanent_uuid: "78b5a2db0bcd41b5a6cb5a29e820ba26" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 34529 } }
I20260620 07:14:16.544126 25618 leader_election.cc:288] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b04da6f74d64797b16e65eab64885be (127.6.53.66:44245), 78b5a2db0bcd41b5a6cb5a29e820ba26 (127.6.53.65:34529)
I20260620 07:14:16.550751 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.544568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:35791 (local address 127.6.53.66:44245)
0620 07:14:16.544780 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:16.544786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:16.544808 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:14:16.545072 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:16.545078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:16.545170 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:16.545270 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:16.545747 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.546694 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.549081 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.549870 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.550008 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.550096 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:16.550377 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 07:14:16.550498 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:14:16.550784 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.544371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:44245 (local address 127.6.53.67:35791)
0620 07:14:16.544681 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:16.544702 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:16.544844 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:16.545330 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:16.545335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:16.545371 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:16.545595 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:16.545609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.546829 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.546833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:16.548900 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:16.548915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.550018 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.550023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:16.550210 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.550232 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:14:16.550351 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 07:14:16.550489 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260620 07:14:16.551421 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.544818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:34179 (local address 127.6.53.65:34529)
0620 07:14:16.545086 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:16.545094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:16.545520 (+ 426us) server_negotiation.cc:407] Connection header received
0620 07:14:16.545701 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:16.545706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:16.545792 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:16.545896 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:16.546462 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.547232 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.549495 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.550451 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.550591 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.550677 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:16.551092 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 07:14:16.551224 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:14:16.551520 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.544548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:34529 (local address 127.6.53.67:34179)
0620 07:14:16.545380 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:16.545402 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:16.545559 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:16.545941 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:16.545944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:16.545968 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:16.546306 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:16.546315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.547379 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.547385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:16.549291 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:16.549300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.550633 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.550639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:16.550853 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.550900 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 07:14:16.551045 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 07:14:16.551196 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":477,"threads_started":1}
I20260620 07:14:16.551955 25418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39041d0e54b9461694a69a41d5f1af57" candidate_uuid: "37ab839d121943169982aab4e19b127c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b04da6f74d64797b16e65eab64885be" is_pre_election: true
I20260620 07:14:16.552529 25418 raft_consensus.cc:2454] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ab839d121943169982aab4e19b127c in term 0.
I20260620 07:14:16.552539 25287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39041d0e54b9461694a69a41d5f1af57" candidate_uuid: "37ab839d121943169982aab4e19b127c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b5a2db0bcd41b5a6cb5a29e820ba26" is_pre_election: true
I20260620 07:14:16.553046 25287 raft_consensus.cc:2454] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ab839d121943169982aab4e19b127c in term 0.
I20260620 07:14:16.553263 25482 leader_election.cc:302] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ab839d121943169982aab4e19b127c, 3b04da6f74d64797b16e65eab64885be; no voters:
I20260620 07:14:16.553653 25618 raft_consensus.cc:2781] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:14:16.553793 25618 raft_consensus.cc:482] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:14:16.553894 25618 raft_consensus.cc:3037] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:16.556037 25618 raft_consensus.cc:504] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b04da6f74d64797b16e65eab64885be" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44245 } } peers { permanent_uuid: "37ab839d121943169982aab4e19b127c" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34745 } } peers { permanent_uuid: "78b5a2db0bcd41b5a6cb5a29e820ba26" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 34529 } }
I20260620 07:14:16.556613 25618 leader_election.cc:288] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [CANDIDATE]: Term 1 election: Requested vote from peers 3b04da6f74d64797b16e65eab64885be (127.6.53.66:44245), 78b5a2db0bcd41b5a6cb5a29e820ba26 (127.6.53.65:34529)
I20260620 07:14:16.556949 25418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39041d0e54b9461694a69a41d5f1af57" candidate_uuid: "37ab839d121943169982aab4e19b127c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b04da6f74d64797b16e65eab64885be"
I20260620 07:14:16.557160 25418 raft_consensus.cc:3037] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:16.557132 25287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39041d0e54b9461694a69a41d5f1af57" candidate_uuid: "37ab839d121943169982aab4e19b127c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b5a2db0bcd41b5a6cb5a29e820ba26"
I20260620 07:14:16.557319 25287 raft_consensus.cc:3037] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:16.559427 25418 raft_consensus.cc:2454] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ab839d121943169982aab4e19b127c in term 1.
I20260620 07:14:16.559681 25287 raft_consensus.cc:2454] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ab839d121943169982aab4e19b127c in term 1.
I20260620 07:14:16.559823 25482 leader_election.cc:302] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ab839d121943169982aab4e19b127c, 3b04da6f74d64797b16e65eab64885be; no voters:
I20260620 07:14:16.560118 25618 raft_consensus.cc:2781] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:16.560556 25618 raft_consensus.cc:686] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [term 1 LEADER]: Becoming Leader. State: Replica: 37ab839d121943169982aab4e19b127c, State: Running, Role: LEADER
I20260620 07:14:16.560992 25618 consensus_queue.cc:237] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b04da6f74d64797b16e65eab64885be" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44245 } } peers { permanent_uuid: "37ab839d121943169982aab4e19b127c" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34745 } } peers { permanent_uuid: "78b5a2db0bcd41b5a6cb5a29e820ba26" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 34529 } }
W20260620 07:14:16.565768 25594 tablet.cc:2369] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:14:16.565701 25146 catalog_manager.cc:5495] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c reported cstate change: term changed from 0 to 1, leader changed from <none> to 37ab839d121943169982aab4e19b127c (127.6.53.67). New cstate: current_term: 1 leader_uuid: "37ab839d121943169982aab4e19b127c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b04da6f74d64797b16e65eab64885be" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37ab839d121943169982aab4e19b127c" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b5a2db0bcd41b5a6cb5a29e820ba26" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 34529 } health_report { overall_health: UNKNOWN } } }
W20260620 07:14:16.618408 25464 tablet.cc:2369] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:14:16.620148 25607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.614500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56126 (local address 127.6.53.67:34745)
0620 07:14:16.614686 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:16.614690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:16.614870 (+ 180us) server_negotiation.cc:407] Connection header received
0620 07:14:16.615036 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:16.615040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:16.615134 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:16.615250 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:14:16.615566 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.616446 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.617920 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.618384 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.618522 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.619113 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:14:16.619582 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:14:16.619756 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:16.619882 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 07:14:16.619924 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:14:16.627664 25287 raft_consensus.cc:1261] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26 [term 1 FOLLOWER]: Refusing update from remote peer 37ab839d121943169982aab4e19b127c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:16.627665 25418 raft_consensus.cc:1261] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be [term 1 FOLLOWER]: Refusing update from remote peer 37ab839d121943169982aab4e19b127c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:16.628531 25621 consensus_queue.cc:1035] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b5a2db0bcd41b5a6cb5a29e820ba26" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 34529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:16.628876 25618 consensus_queue.cc:1035] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b04da6f74d64797b16e65eab64885be" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:16.656018 25145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:16.656347 25145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:16.660197 25145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33210d7457d547f39c82cba10251c3d2: Sending DeleteTablet for 3 replicas of tablet 39041d0e54b9461694a69a41d5f1af57
I20260620 07:14:16.661540 25267 tablet_service.cc:1514] Processing DeleteTablet for tablet 39041d0e54b9461694a69a41d5f1af57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:16 UTC) from {username='slave'} at 127.0.0.1:58920
I20260620 07:14:16.661513 25398 tablet_service.cc:1514] Processing DeleteTablet for tablet 39041d0e54b9461694a69a41d5f1af57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:16 UTC) from {username='slave'} at 127.0.0.1:45062
I20260620 07:14:16.661535 25528 tablet_service.cc:1514] Processing DeleteTablet for tablet 39041d0e54b9461694a69a41d5f1af57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:16 UTC) from {username='slave'} at 127.0.0.1:56122
I20260620 07:14:16.663406 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 25205
I20260620 07:14:16.674520 25635 tablet_replica.cc:330] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be: stopping tablet replica
I20260620 07:14:16.675196 25636 tablet_replica.cc:330] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26: stopping tablet replica
I20260620 07:14:16.675824 25636 raft_consensus.cc:2229] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:16.676366 25635 raft_consensus.cc:2229] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:16.676949 25636 raft_consensus.cc:2258] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:16.677430 25635 raft_consensus.cc:2258] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:16.679778 25636 ts_tablet_manager.cc:1905] T 39041d0e54b9461694a69a41d5f1af57 P 78b5a2db0bcd41b5a6cb5a29e820ba26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:16.681478 25635 ts_tablet_manager.cc:1905] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:16.682138 25634 tablet_replica.cc:330] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c: stopping tablet replica
I20260620 07:14:16.683087 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.664600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58928 (local address 127.6.53.65:34529)
0620 07:14:16.664766 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:16.664774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:16.674734 (+ 9960us) server_negotiation.cc:407] Connection header received
0620 07:14:16.675089 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:16.675095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:16.675224 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:16.675339 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:16.675953 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.677047 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.678963 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.679607 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.679722 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.681106 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:16.681116 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:16.681132 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:16.681866 (+ 734us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:16.681925 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:16.681949 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:16.682065 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:16.682605 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 07:14:16.682717 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:14:16.683614 25634 raft_consensus.cc:2229] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:14:16.683869 25327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:16.685904 25634 raft_consensus.cc:2258] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:16.691830 25635 ts_tablet_manager.cc:1918] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:16.692008 25635 log.cc:1196] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/39041d0e54b9461694a69a41d5f1af57
I20260620 07:14:16.692466 25635 ts_tablet_manager.cc:1939] T 39041d0e54b9461694a69a41d5f1af57 P 3b04da6f74d64797b16e65eab64885be: Deleting consensus metadata
I20260620 07:14:16.694046 25131 catalog_manager.cc:4841] TS 3b04da6f74d64797b16e65eab64885be (127.6.53.66:44245): tablet 39041d0e54b9461694a69a41d5f1af57 (table test-table [id=5f6fa17003a4435195b4dca96be2e7df]) successfully deleted
I20260620 07:14:16.694836 25634 ts_tablet_manager.cc:1905] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:16.699885 25634 ts_tablet_manager.cc:1918] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:16.700069 25634 log.cc:1196] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/39041d0e54b9461694a69a41d5f1af57
I20260620 07:14:16.700497 25634 ts_tablet_manager.cc:1939] T 39041d0e54b9461694a69a41d5f1af57 P 37ab839d121943169982aab4e19b127c: Deleting consensus metadata
I20260620 07:14:16.701857 25133 catalog_manager.cc:4841] TS 37ab839d121943169982aab4e19b127c (127.6.53.67:34745): tablet 39041d0e54b9461694a69a41d5f1af57 (table test-table [id=5f6fa17003a4435195b4dca96be2e7df]) successfully deleted
I20260620 07:14:16.804967 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 25205
I20260620 07:14:16.822847 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 25336
W20260620 07:14:16.822899 25132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.53.65:34529 (error 108)
I20260620 07:14:16.824892 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.824359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:34529 (local address 127.0.0.1:58940)
0620 07:14:16.824608 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:16.824728 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260620 07:14:16.825121 25132 catalog_manager.cc:4584] TS 78b5a2db0bcd41b5a6cb5a29e820ba26 (127.6.53.65:34529): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39041d0e54b9461694a69a41d5f1af57: Network error: Client connection negotiation failed: client connection to 127.6.53.65:34529: connect: Connection refused (error 111)
I20260620 07:14:16.830578 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.823980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.6.53.66:44245)
0620 07:14:16.824156 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:16.824163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:16.824186 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:14:16.824239 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:16.824242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:16.824332 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:16.824452 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:16.824833 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.825707 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.827003 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.827536 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.827670 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.828898 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:16.828911 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:16.828930 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:16.829674 (+ 744us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:16.829731 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:16.829761 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:16.829874 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:16.830074 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 07:14:16.830191 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:14:16.834681 25458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:16.878713 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.878138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:34529 (local address 127.0.0.1:58948)
0620 07:14:16.878419 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:16.878513 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260620 07:14:16.879765 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.879373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:34529 (local address 127.0.0.1:58962)
0620 07:14:16.879543 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:16.879616 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 07:14:16.927613 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 25336
I20260620 07:14:16.940157 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.939741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:34529 (local address 127.0.0.1:58970)
0620 07:14:16.939932 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:16.940008 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 07:14:16.941103 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.940786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:34529 (local address 127.0.0.1:58972)
0620 07:14:16.940932 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:16.940995 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 07:14:16.950681 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 25466
I20260620 07:14:16.958031 25607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:16.951384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56134 (local address 127.6.53.67:34745)
0620 07:14:16.951560 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:16.951565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:16.951581 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:14:16.951759 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:16.951763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:16.951835 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:16.951914 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:16.952386 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.953147 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.954920 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:16.955350 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:16.955439 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:16.956589 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:16.956598 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:16.956611 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:16.957221 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:16.957268 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:16.957287 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:16.957367 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:16.957544 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 07:14:16.957628 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:14:16.958732 25588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:17.021613 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:17.021168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:34529 (local address 127.0.0.1:58986)
0620 07:14:17.021369 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:17.021452 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 07:14:17.022611 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:17.022216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:34529 (local address 127.0.0.1:59000)
0620 07:14:17.022440 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:17.022513 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260620 07:14:17.076506 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 25466
I20260620 07:14:17.096036 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 25115
I20260620 07:14:17.097076 25176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:17.202271 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 25115
2026-06-20T07:14:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2830 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:17 dist-test-slave-4n6h krb5kdc[25645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:17 dist-test-slave-4n6h krb5kdc[25645](info): set up 2 sockets
Jun 20 07:14:17 dist-test-slave-4n6h krb5kdc[25645](info): commencing operation
krb5kdc: starting...
W20260620 07:14:18.302359 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:18 dist-test-slave-4n6h krb5kdc[25645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:14:18.533210 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:41517
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:41517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:18.856964 25658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:18.857404 25658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:18.857590 25658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:18.867753 25658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:18.867877 25658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:18.867976 25658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:18.868057 25658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:18.868119 25658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:18.877961 25658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:41517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:41517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:18.879258 25658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:18.880530 25658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:18.892068 25664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:18.892287 25663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:18.892310 25666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:18.893019 25658 server_base.cc:1034] running on GCE node
I20260620 07:14:18.893784 25658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:18.895226 25658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:18.896471 25658 hybrid_clock.cc:648] HybridClock initialized: now 1781939658896431 us; error 58 us; skew 500 ppm
I20260620 07:14:18.896822 25658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:18 dist-test-slave-4n6h krb5kdc[25645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939658, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:18.899868 25658 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:14:18.901335 25658 webserver.cc:460] Webserver started at http://127.6.53.126:42817/ using document root <none> and password file <none>
I20260620 07:14:18.901989 25658 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:18.902127 25658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:18.902493 25658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:18.905303 25658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "9c229a8ed98a4c43982583ffc0708ba6"
format_stamp: "Formatted at 2026-06-20 07:14:18 on dist-test-slave-4n6h"
server_key: "60314b66fdcd61f5ddfc03593e93cd61"
server_key_iv: "d5eb81b43ed413e7fa439ac47006e05c"
server_key_version: "encryptionkey@0"
I20260620 07:14:18.906039 25658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "9c229a8ed98a4c43982583ffc0708ba6"
format_stamp: "Formatted at 2026-06-20 07:14:18 on dist-test-slave-4n6h"
server_key: "60314b66fdcd61f5ddfc03593e93cd61"
server_key_iv: "d5eb81b43ed413e7fa439ac47006e05c"
server_key_version: "encryptionkey@0"
I20260620 07:14:18.910952 25658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 07:14:18.914145 25673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:18.915340 25658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:14:18.915514 25658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "9c229a8ed98a4c43982583ffc0708ba6"
format_stamp: "Formatted at 2026-06-20 07:14:18 on dist-test-slave-4n6h"
server_key: "60314b66fdcd61f5ddfc03593e93cd61"
server_key_iv: "d5eb81b43ed413e7fa439ac47006e05c"
server_key_version: "encryptionkey@0"
I20260620 07:14:18.915699 25658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:18.931593 25658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:18.935215 25658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:18.935477 25658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:18.953043 25658 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:41517
I20260620 07:14:18.953078 25724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:41517 every 8 connection(s)
I20260620 07:14:18.954300 25658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:18.957626 25725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:18.962162 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 25658
I20260620 07:14:18.962363 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:18.962918 6357 external_mini_cluster.cc:1428] Setting key 4a1b614cd7e74bdff7d6297314b9e74b
I20260620 07:14:18.968393 25725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6: Bootstrap starting.
I20260620 07:14:18.971819 25725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:18.972903 25725 log.cc:824] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:18.976094 25725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6: No bootstrap required, opened a new log
Jun 20 07:14:18 dist-test-slave-4n6h krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:14:18.983548 25725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c229a8ed98a4c43982583ffc0708ba6" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41517 } }
I20260620 07:14:18.983891 25725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:18.984021 25725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c229a8ed98a4c43982583ffc0708ba6, State: Initialized, Role: FOLLOWER
I20260620 07:14:18.984680 25725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c229a8ed98a4c43982583ffc0708ba6" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41517 } }
I20260620 07:14:18.984954 25725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:18.985087 25725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:18.985282 25725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:18.986197 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:18.964597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.6.53.126:41517)
0620 07:14:18.965380 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:18.965391 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:18.965430 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:14:18.967166 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:18.967186 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:18.967509 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:18.967783 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:18.969513 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:18.970845 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:18.972522 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:18.973195 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:18.973338 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:18.978323 (+ 4985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:18.978332 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:18.978349 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:18.978408 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:18.981664 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:18.982546 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:18.982553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:18.982559 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:18.982741 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:18.983184 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:18.983192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:18.983195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:18.983707 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:18.984044 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:18.984461 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 07:14:18.984884 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":380,"threads_started":1}
I20260620 07:14:18.987849 25725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c229a8ed98a4c43982583ffc0708ba6" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41517 } }
I20260620 07:14:18.988309 25725 leader_election.cc:302] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c229a8ed98a4c43982583ffc0708ba6; no voters:
I20260620 07:14:18.988941 25725 leader_election.cc:288] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:18.989045 25730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:18.989722 25730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c229a8ed98a4c43982583ffc0708ba6, State: Running, Role: LEADER
I20260620 07:14:18.990325 25730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c229a8ed98a4c43982583ffc0708ba6" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41517 } }
I20260620 07:14:18.991117 25725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:18.993024 25731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c229a8ed98a4c43982583ffc0708ba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c229a8ed98a4c43982583ffc0708ba6" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41517 } } }
I20260620 07:14:18.993383 25731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:18.993829 25732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c229a8ed98a4c43982583ffc0708ba6. Latest consensus state: current_term: 1 leader_uuid: "9c229a8ed98a4c43982583ffc0708ba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c229a8ed98a4c43982583ffc0708ba6" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 41517 } } }
I20260620 07:14:18.994052 25732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6 [sys.catalog]: This master's current role is: LEADER
W20260620 07:14:19.001145 25746 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:14:19.001281 25746 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:14:19.001443 25738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:19.003585 25738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:19.011261 25738 catalog_manager.cc:1347] Generated new cluster ID: cd303b182f8e4451a2b6b1b79ba6876a
I20260620 07:14:19.011379 25738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:19.019841 25738 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:19.021296 25738 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:19.027324 25738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6: Generated new TSK 0
I20260620 07:14:19.028143 25738 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
Entry for principal kudu/127.6.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab.
I20260620 07:14:19.101485 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41517
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:19.454952 25751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:19.455214 25751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:19.455307 25751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:19.462423 25751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:19.462517 25751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:19.462621 25751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:14:19.471868 25751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.65.keytab
--principal=kudu/127.6.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:19.472991 25751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:19.474228 25751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:19.486488 25757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:19.486699 25759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:19.487287 25756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:19.488205 25751 server_base.cc:1034] running on GCE node
I20260620 07:14:19.488801 25751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:19.489732 25751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:19.490942 25751 hybrid_clock.cc:648] HybridClock initialized: now 1781939659490897 us; error 24 us; skew 500 ppm
I20260620 07:14:19.491307 25751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:19 dist-test-slave-4n6h krb5kdc[25645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939659, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:19.494583 25751 init.cc:377] Logged in from keytab as kudu/127.6.53.65@KRBTEST.COM (short username kudu)
I20260620 07:14:19.496150 25751 webserver.cc:460] Webserver started at http://127.6.53.65:45129/ using document root <none> and password file <none>
I20260620 07:14:19.496816 25751 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:19.496971 25751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:19.497285 25751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:19.500461 25751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "982427670b414556824d266b14d3a10d"
format_stamp: "Formatted at 2026-06-20 07:14:19 on dist-test-slave-4n6h"
server_key: "84b5119acb1d16874d8be02f15ce3792"
server_key_iv: "3120c6a8cffcfb9908e517dbb13a8d8f"
server_key_version: "encryptionkey@0"
I20260620 07:14:19.501367 25751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "982427670b414556824d266b14d3a10d"
format_stamp: "Formatted at 2026-06-20 07:14:19 on dist-test-slave-4n6h"
server_key: "84b5119acb1d16874d8be02f15ce3792"
server_key_iv: "3120c6a8cffcfb9908e517dbb13a8d8f"
server_key_version: "encryptionkey@0"
I20260620 07:14:19.506567 25751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:14:19.509955 25766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:19.511264 25751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:14:19.511483 25751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "982427670b414556824d266b14d3a10d"
format_stamp: "Formatted at 2026-06-20 07:14:19 on dist-test-slave-4n6h"
server_key: "84b5119acb1d16874d8be02f15ce3792"
server_key_iv: "3120c6a8cffcfb9908e517dbb13a8d8f"
server_key_version: "encryptionkey@0"
I20260620 07:14:19.511708 25751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:19.535526 25751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:19.540225 25751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:19.540562 25751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:19.541595 25751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:19.543004 25751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:19.543116 25751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:19.543272 25751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:19.543346 25751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:19.576817 25751 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:38861
I20260620 07:14:19.576864 25878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:38861 every 8 connection(s)
I20260620 07:14:19.578035 25751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:14:19.580024 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 25751
I20260620 07:14:19.580201 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:14:19.580848 6357 external_mini_cluster.cc:1428] Setting key ae9f3bb0e1373cad67a1ca053fe41db8
Jun 20 07:14:19 dist-test-slave-4n6h krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939659, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.65@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:14:19.598410 25881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:19.580636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:48369 (local address 127.6.53.126:41517)
0620 07:14:19.581163 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:19.581167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:19.582298 (+ 1131us) server_negotiation.cc:407] Connection header received
0620 07:14:19.584130 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:19.584139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:19.584228 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:19.584341 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:19.585750 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:19.586901 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:19.588410 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:19.589118 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:19.589257 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:19.592668 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:19.592673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:19.592676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:19.592706 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:19.595373 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:19.596174 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:19.596181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:19.596185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:19.596343 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:19.596750 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:19.596758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:19.596761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:19.597015 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:19.597163 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:19.597799 (+ 636us) server_negotiation.cc:299] Negotiation successful
0620 07:14:19.598035 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":223,"threads_started":1}
I20260620 07:14:19.599310 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:19.581256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41517 (local address 127.6.53.65:48369)
0620 07:14:19.582114 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:19.582157 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:19.583861 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:19.584527 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:19.584534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:19.585065 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:19.585579 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:19.585602 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:19.587055 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:19.587059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:19.588265 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:19.588276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:19.589314 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:19.589318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:19.589506 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:19.590165 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:19.590186 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:19.592458 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:19.595503 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:19.595511 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:19.595526 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:19.596056 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:19.596448 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:19.596454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:19.596457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:19.596645 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:19.597143 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:19.597150 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:19.597453 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 07:14:19.597702 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 07:14:19.598206 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":383,"threads_started":1}
I20260620 07:14:19.600858 25879 heartbeater.cc:347] Connected to a master server at 127.6.53.126:41517
I20260620 07:14:19.601161 25879 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:19.601853 25879 heartbeater.cc:511] Master 127.6.53.126:41517 requested a full tablet report, sending...
I20260620 07:14:19.603827 25690 ts_manager.cc:194] Registered new tserver with Master: 982427670b414556824d266b14d3a10d (127.6.53.65:38861)
WARNING: no policy specified for kudu/127.6.53.66@KRBTEST.COM; defaulting to no policy
I20260620 07:14:19.605257 25690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.65@KRBTEST.COM'} at 127.6.53.65:48369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
Entry for principal kudu/127.6.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab.
I20260620 07:14:19.633189 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41517
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:19.949421 25885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:19.949679 25885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:19.949770 25885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:19.956972 25885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:19.957072 25885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:19.957262 25885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:14:19.966650 25885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.66.keytab
--principal=kudu/127.6.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:19.967729 25885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:19.968892 25885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:19.981587 25890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:19.981499 25893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:19.981630 25891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:19.982424 25885 server_base.cc:1034] running on GCE node
I20260620 07:14:19.982993 25885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:19.983922 25885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:19.985131 25885 hybrid_clock.cc:648] HybridClock initialized: now 1781939659985086 us; error 44 us; skew 500 ppm
I20260620 07:14:19.985411 25885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:19 dist-test-slave-4n6h krb5kdc[25645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939659, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:19.988466 25885 init.cc:377] Logged in from keytab as kudu/127.6.53.66@KRBTEST.COM (short username kudu)
I20260620 07:14:19.989825 25885 webserver.cc:460] Webserver started at http://127.6.53.66:33161/ using document root <none> and password file <none>
I20260620 07:14:19.990504 25885 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:19.990638 25885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:19.990955 25885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:19.993728 25885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "da6034a055694fea8b554b4a98b35c92"
format_stamp: "Formatted at 2026-06-20 07:14:19 on dist-test-slave-4n6h"
server_key: "9775b057b96df8a72ab617b7e846bbed"
server_key_iv: "71366b373389bc1f9cba197f454776e1"
server_key_version: "encryptionkey@0"
I20260620 07:14:19.994563 25885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "da6034a055694fea8b554b4a98b35c92"
format_stamp: "Formatted at 2026-06-20 07:14:19 on dist-test-slave-4n6h"
server_key: "9775b057b96df8a72ab617b7e846bbed"
server_key_iv: "71366b373389bc1f9cba197f454776e1"
server_key_version: "encryptionkey@0"
I20260620 07:14:19.999440 25885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 07:14:20.002805 25900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:20.004086 25885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:14:20.004242 25885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "da6034a055694fea8b554b4a98b35c92"
format_stamp: "Formatted at 2026-06-20 07:14:19 on dist-test-slave-4n6h"
server_key: "9775b057b96df8a72ab617b7e846bbed"
server_key_iv: "71366b373389bc1f9cba197f454776e1"
server_key_version: "encryptionkey@0"
I20260620 07:14:20.004441 25885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:20.033605 25885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:20.037751 25885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:20.038034 25885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:20.038981 25885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:20.040246 25885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:20.040340 25885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:20.040482 25885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:20.040551 25885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:20.073166 25885 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:45449
I20260620 07:14:20.073248 26012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:45449 every 8 connection(s)
I20260620 07:14:20.074455 25885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:14:20.075305 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 25885
I20260620 07:14:20.075517 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:14:20.076146 6357 external_mini_cluster.cc:1428] Setting key bd5f9a7d9347d28d009c3d9dc26c91c7
Jun 20 07:14:20 dist-test-slave-4n6h krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939659, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.66@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:14:20.096498 25881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.077350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:51443 (local address 127.6.53.126:41517)
0620 07:14:20.077520 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.077524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.078964 (+ 1440us) server_negotiation.cc:407] Connection header received
0620 07:14:20.080793 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.080803 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.080912 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.081022 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:20.082440 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.083591 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.085064 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.085826 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.085979 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.089839 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:20.089847 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:20.089851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:20.089890 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:20.093077 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:20.094015 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:20.094019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:20.094021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:20.094137 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:20.094671 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:20.094674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:20.094676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:20.094954 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:20.095064 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.096014 (+ 950us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.096216 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:14:20.097855 26015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.077950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41517 (local address 127.6.53.66:51443)
0620 07:14:20.078806 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:20.078847 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:20.080538 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:20.081205 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:20.081213 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:20.081703 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:20.082211 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.082226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.083734 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.083737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.084921 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.084932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.086034 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.086038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.086218 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.086946 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:20.086967 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:20.089567 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:20.093219 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:20.093232 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:20.093250 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:20.093876 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:20.094245 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:20.094275 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:20.094282 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:20.094561 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:20.095069 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:20.095080 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:20.095530 (+ 450us) client_negotiation.cc:769] Sending connection context
0620 07:14:20.095831 (+ 301us) client_negotiation.cc:240] Negotiation successful
0620 07:14:20.096435 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":402,"threads_started":1}
I20260620 07:14:20.099560 26013 heartbeater.cc:347] Connected to a master server at 127.6.53.126:41517
I20260620 07:14:20.099893 26013 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:20.100692 26013 heartbeater.cc:511] Master 127.6.53.126:41517 requested a full tablet report, sending...
I20260620 07:14:20.102319 25690 ts_manager.cc:194] Registered new tserver with Master: da6034a055694fea8b554b4a98b35c92 (127.6.53.66:45449)
I20260620 07:14:20.103541 25690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.66@KRBTEST.COM'} at 127.6.53.66:51443
WARNING: no policy specified for kudu/127.6.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
Entry for principal kudu/127.6.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab.
I20260620 07:14:20.133435 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41517
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:20.450932 26018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:20.451205 26018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:20.451361 26018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:20.458446 26018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:20.458561 26018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:20.458732 26018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:14:20.468170 26018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.67.keytab
--principal=kudu/127.6.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:41517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:20.469343 26018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:20.470582 26018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:20.482872 26024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:20.482961 26023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:20.482915 26026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:20.484760 26018 server_base.cc:1034] running on GCE node
I20260620 07:14:20.485271 26018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:20.486089 26018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:20.487290 26018 hybrid_clock.cc:648] HybridClock initialized: now 1781939660487234 us; error 52 us; skew 500 ppm
I20260620 07:14:20.487562 26018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:20 dist-test-slave-4n6h krb5kdc[25645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939660, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:20.490581 26018 init.cc:377] Logged in from keytab as kudu/127.6.53.67@KRBTEST.COM (short username kudu)
I20260620 07:14:20.491894 26018 webserver.cc:460] Webserver started at http://127.6.53.67:33953/ using document root <none> and password file <none>
I20260620 07:14:20.492548 26018 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:20.492643 26018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:20.492882 26018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:20.495770 26018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "d017c0c3ce9740f09e8c731aa9a019f3"
format_stamp: "Formatted at 2026-06-20 07:14:20 on dist-test-slave-4n6h"
server_key: "69216852452d0694cb920b923210ff17"
server_key_iv: "1c5f38fa2df6d007ef6c54ec2457ae34"
server_key_version: "encryptionkey@0"
I20260620 07:14:20.496539 26018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "d017c0c3ce9740f09e8c731aa9a019f3"
format_stamp: "Formatted at 2026-06-20 07:14:20 on dist-test-slave-4n6h"
server_key: "69216852452d0694cb920b923210ff17"
server_key_iv: "1c5f38fa2df6d007ef6c54ec2457ae34"
server_key_version: "encryptionkey@0"
I20260620 07:14:20.501147 26018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:14:20.504459 26033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:20.505630 26018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 07:14:20.505744 26018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "d017c0c3ce9740f09e8c731aa9a019f3"
format_stamp: "Formatted at 2026-06-20 07:14:20 on dist-test-slave-4n6h"
server_key: "69216852452d0694cb920b923210ff17"
server_key_iv: "1c5f38fa2df6d007ef6c54ec2457ae34"
server_key_version: "encryptionkey@0"
I20260620 07:14:20.505898 26018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:20.521580 26018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:20.525375 26018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:20.525642 26018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:20.526593 26018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:20.527796 26018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:20.527868 26018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:20.528026 26018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:20.528081 26018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:20.559568 26018 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:34175
I20260620 07:14:20.559654 26145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:34175 every 8 connection(s)
I20260620 07:14:20.560741 26018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:14:20.564414 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 26018
I20260620 07:14:20.564584 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:14:20.565040 6357 external_mini_cluster.cc:1428] Setting key 430b42786f072cbee1b821b8183ad53d
Jun 20 07:14:20 dist-test-slave-4n6h krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939660, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.67@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:14:20.581211 26148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.563614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:41517 (local address 127.6.53.67:56457)
0620 07:14:20.564311 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:20.564347 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:20.566395 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:20.567104 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:20.567112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:20.567583 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:20.568081 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.568097 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.569153 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.569158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.570415 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.570427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.571165 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.571174 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.571402 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.571999 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:20.572019 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:20.574044 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:20.577095 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:20.577103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:20.577116 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:20.577522 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:20.577926 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:20.577931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:20.577935 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:20.578192 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:20.578870 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:20.578877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:20.579250 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 07:14:20.579527 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 07:14:20.579998 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":282,"threads_started":1}
I20260620 07:14:20.582279 25881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.563263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:56457 (local address 127.6.53.126:41517)
0620 07:14:20.563467 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.563471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.564503 (+ 1032us) server_negotiation.cc:407] Connection header received
0620 07:14:20.566650 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.566656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.566772 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.566881 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:20.568245 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.569001 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.570554 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.570958 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.571090 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.574244 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:20.574289 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:20.574295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:20.574337 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:20.576955 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:20.577646 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:20.577652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:20.577656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:20.577794 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:20.578352 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:20.578357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:20.578360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:20.578733 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:20.578895 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.581696 (+ 2801us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.581909 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:14:20.583204 26146 heartbeater.cc:347] Connected to a master server at 127.6.53.126:41517
I20260620 07:14:20.583438 26146 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:20.584033 26146 heartbeater.cc:511] Master 127.6.53.126:41517 requested a full tablet report, sending...
I20260620 07:14:20.585263 25690 ts_manager.cc:194] Registered new tserver with Master: d017c0c3ce9740f09e8c731aa9a019f3 (127.6.53.67:34175)
I20260620 07:14:20.586072 25690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.53.67@KRBTEST.COM'} at 127.6.53.67:56457
I20260620 07:14:20.591795 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:14:20.608028 25881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.595873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.6.53.126:41517)
0620 07:14:20.596026 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.596030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.596489 (+ 459us) server_negotiation.cc:407] Connection header received
0620 07:14:20.596619 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.596622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.596676 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.596752 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:20.598050 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.598881 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.600128 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.600493 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.600586 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.602910 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:20.602914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:20.602918 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:20.602945 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:20.605446 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:20.606334 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:20.606341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:20.606346 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:20.606501 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:20.606839 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:20.606842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:20.606844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:20.607135 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:20.607379 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.607644 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.607799 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:14:20.608289 25879 heartbeater.cc:503] Master 127.6.53.126:41517 was elected leader, sending a full tablet report...
I20260620 07:14:20.611938 25689 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:14:20.618868 25689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:14:20.639770 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.629702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:34175 (local address 127.0.0.1:58196)
0620 07:14:20.630759 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:20.630806 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:20.631017 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:20.632236 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:20.632243 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:20.632273 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:20.632726 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.632744 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.634935 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.634941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.637154 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.637168 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.638802 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.638808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.639002 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.639052 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 07:14:20.639200 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 07:14:20.639379 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":590,"threads_started":1}
I20260620 07:14:20.639784 26159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.630067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58196 (local address 127.6.53.67:34175)
0620 07:14:20.631356 (+ 1289us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.631364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.631395 (+ 31us) server_negotiation.cc:407] Connection header received
0620 07:14:20.631487 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.631496 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.631735 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.631919 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:20.633154 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.634795 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.637345 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.638617 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.638750 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.638987 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.639259 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.639441 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1101,"thread_start_us":937,"threads_started":1}
I20260620 07:14:20.640782 26161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.631495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58098 (local address 127.6.53.66:45449)
0620 07:14:20.632855 (+ 1360us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.632865 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.632890 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:14:20.632982 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.632990 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.633185 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.633917 (+ 732us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:20.634154 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.635389 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.637819 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.639218 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.640060 (+ 842us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.640252 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.640312 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.640498 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1201,"thread_start_us":1113,"threads_started":1}
I20260620 07:14:20.641378 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.631060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:45449 (local address 127.0.0.1:58098)
0620 07:14:20.631545 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:20.631566 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:20.631685 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:20.633393 (+ 1708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:20.633401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:20.633425 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:20.633822 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.633835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.635513 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.635521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.637649 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.637663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.639427 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.639442 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.639628 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.639662 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 07:14:20.639728 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 07:14:20.639865 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":1,"thread_start_us":230,"threads_started":1}
I20260620 07:14:20.641522 26163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.632152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.6.53.65:38861)
0620 07:14:20.633211 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.633221 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.633248 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:14:20.633338 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.633347 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.633560 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.633955 (+ 395us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:20.634515 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.635909 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.638213 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.639474 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.639614 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.639820 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.640835 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.641181 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":727,"threads_started":1}
I20260620 07:14:20.641939 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.631615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38861 (local address 127.0.0.1:42840)
0620 07:14:20.632216 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:20.632237 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:20.632353 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:20.633782 (+ 1429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:20.633790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:20.633817 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:20.634363 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.634372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.636061 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.636066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.638065 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.638082 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.640277 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.640281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.640422 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.640451 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 07:14:20.640538 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 07:14:20.640683 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":748,"threads_started":1}
I20260620 07:14:20.643797 26081 tablet_service.cc:1467] Processing CreateTablet for tablet 64f39d3d5e3341bba73d3a3de692d2b5 (DEFAULT_TABLE table=test-table [id=5bd7604dfbe94ff698cdf68cb262c9ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:20.644618 25814 tablet_service.cc:1467] Processing CreateTablet for tablet 64f39d3d5e3341bba73d3a3de692d2b5 (DEFAULT_TABLE table=test-table [id=5bd7604dfbe94ff698cdf68cb262c9ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:20.644682 25948 tablet_service.cc:1467] Processing CreateTablet for tablet 64f39d3d5e3341bba73d3a3de692d2b5 (DEFAULT_TABLE table=test-table [id=5bd7604dfbe94ff698cdf68cb262c9ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:20.645260 26081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f39d3d5e3341bba73d3a3de692d2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:20.645843 25814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f39d3d5e3341bba73d3a3de692d2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:20.645915 25948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f39d3d5e3341bba73d3a3de692d2b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:20.656047 26164 tablet_bootstrap.cc:492] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d: Bootstrap starting.
I20260620 07:14:20.659680 26164 tablet_bootstrap.cc:654] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:20.660645 26164 log.cc:824] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:20.663029 26166 tablet_bootstrap.cc:492] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3: Bootstrap starting.
I20260620 07:14:20.663506 26164 tablet_bootstrap.cc:492] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d: No bootstrap required, opened a new log
I20260620 07:14:20.663758 26164 ts_tablet_manager.cc:1397] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 07:14:20.665711 26166 tablet_bootstrap.cc:654] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:20.665979 26165 tablet_bootstrap.cc:492] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92: Bootstrap starting.
I20260620 07:14:20.666786 26166 log.cc:824] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:20.669154 26165 tablet_bootstrap.cc:654] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:20.670089 26166 tablet_bootstrap.cc:492] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3: No bootstrap required, opened a new log
I20260620 07:14:20.670271 26165 log.cc:824] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:20.670410 26166 ts_tablet_manager.cc:1397] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 07:14:20.673018 26165 tablet_bootstrap.cc:492] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92: No bootstrap required, opened a new log
I20260620 07:14:20.673264 26165 ts_tablet_manager.cc:1397] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260620 07:14:20.672989 26164 raft_consensus.cc:348] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d017c0c3ce9740f09e8c731aa9a019f3" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34175 } } peers { permanent_uuid: "da6034a055694fea8b554b4a98b35c92" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45449 } } peers { permanent_uuid: "982427670b414556824d266b14d3a10d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38861 } }
I20260620 07:14:20.673401 26164 raft_consensus.cc:374] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:20.673497 26164 raft_consensus.cc:729] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982427670b414556824d266b14d3a10d, State: Initialized, Role: FOLLOWER
I20260620 07:14:20.674108 26164 consensus_queue.cc:260] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d017c0c3ce9740f09e8c731aa9a019f3" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34175 } } peers { permanent_uuid: "da6034a055694fea8b554b4a98b35c92" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45449 } } peers { permanent_uuid: "982427670b414556824d266b14d3a10d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38861 } }
I20260620 07:14:20.678551 26166 raft_consensus.cc:348] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d017c0c3ce9740f09e8c731aa9a019f3" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34175 } } peers { permanent_uuid: "da6034a055694fea8b554b4a98b35c92" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45449 } } peers { permanent_uuid: "982427670b414556824d266b14d3a10d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38861 } }
I20260620 07:14:20.678994 26166 raft_consensus.cc:374] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:20.679098 26166 raft_consensus.cc:729] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d017c0c3ce9740f09e8c731aa9a019f3, State: Initialized, Role: FOLLOWER
I20260620 07:14:20.679658 26166 consensus_queue.cc:260] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d017c0c3ce9740f09e8c731aa9a019f3" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34175 } } peers { permanent_uuid: "da6034a055694fea8b554b4a98b35c92" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45449 } } peers { permanent_uuid: "982427670b414556824d266b14d3a10d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38861 } }
I20260620 07:14:20.680182 26164 ts_tablet_manager.cc:1428] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d: Time spent starting tablet: real 0.016s user 0.009s sys 0.008s
I20260620 07:14:20.681111 26146 heartbeater.cc:503] Master 127.6.53.126:41517 was elected leader, sending a full tablet report...
I20260620 07:14:20.681731 26166 ts_tablet_manager.cc:1428] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 07:14:20.684595 26165 raft_consensus.cc:348] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d017c0c3ce9740f09e8c731aa9a019f3" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34175 } } peers { permanent_uuid: "da6034a055694fea8b554b4a98b35c92" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45449 } } peers { permanent_uuid: "982427670b414556824d266b14d3a10d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38861 } }
I20260620 07:14:20.685029 26165 raft_consensus.cc:374] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:20.685163 26165 raft_consensus.cc:729] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6034a055694fea8b554b4a98b35c92, State: Initialized, Role: FOLLOWER
I20260620 07:14:20.685837 26165 consensus_queue.cc:260] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d017c0c3ce9740f09e8c731aa9a019f3" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34175 } } peers { permanent_uuid: "da6034a055694fea8b554b4a98b35c92" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45449 } } peers { permanent_uuid: "982427670b414556824d266b14d3a10d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38861 } }
I20260620 07:14:20.686843 26013 heartbeater.cc:503] Master 127.6.53.126:41517 was elected leader, sending a full tablet report...
I20260620 07:14:20.687393 26165 ts_tablet_manager.cc:1428] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
W20260620 07:14:20.812331 26147 tablet.cc:2369] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:14:20.826671 26014 tablet.cc:2369] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:14:20.830317 25880 tablet.cc:2369] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:14:20.880522 26172 raft_consensus.cc:482] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:20.880800 26172 raft_consensus.cc:504] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d017c0c3ce9740f09e8c731aa9a019f3" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34175 } } peers { permanent_uuid: "da6034a055694fea8b554b4a98b35c92" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45449 } } peers { permanent_uuid: "982427670b414556824d266b14d3a10d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38861 } }
I20260620 07:14:20.882411 26172 leader_election.cc:288] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d017c0c3ce9740f09e8c731aa9a019f3 (127.6.53.67:34175), 982427670b414556824d266b14d3a10d (127.6.53.65:38861)
I20260620 07:14:20.889060 26159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.882738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:52373 (local address 127.6.53.67:34175)
0620 07:14:20.882923 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.882929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.883356 (+ 427us) server_negotiation.cc:407] Connection header received
0620 07:14:20.883724 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.883733 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.883829 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.883889 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:20.884568 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.885641 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.887425 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.888175 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.888323 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.888430 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.888689 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.888812 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:14:20.889154 26173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.882639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:34175 (local address 127.6.53.66:52373)
0620 07:14:20.883259 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:20.883277 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:20.883488 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:20.884084 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:20.884087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:20.884111 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:20.884437 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.884445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.885753 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.885756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.887263 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.887275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.888381 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.888387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.888536 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.888559 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:14:20.888663 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 07:14:20.888788 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":299,"threads_started":1}
I20260620 07:14:20.889561 26163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.883697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:57039 (local address 127.6.53.65:38861)
0620 07:14:20.883914 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.883920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.884240 (+ 320us) server_negotiation.cc:407] Connection header received
0620 07:14:20.884464 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.884468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.884557 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.884653 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:20.885099 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.886038 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.887818 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.888584 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.888735 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.888843 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.889197 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.889336 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:14:20.890233 26174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.883339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:38861 (local address 127.6.53.66:57039)
0620 07:14:20.884117 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:20.884131 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:20.884308 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:20.884665 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:20.884668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:20.884685 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:20.884947 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.884954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.886175 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.886191 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.887661 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.887672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.888815 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.888821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.888993 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.889014 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 07:14:20.889776 (+ 762us) client_negotiation.cc:240] Negotiation successful
0620 07:14:20.889907 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":454,"threads_started":1}
I20260620 07:14:20.890120 26101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f39d3d5e3341bba73d3a3de692d2b5" candidate_uuid: "da6034a055694fea8b554b4a98b35c92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d017c0c3ce9740f09e8c731aa9a019f3" is_pre_election: true
I20260620 07:14:20.890712 26101 raft_consensus.cc:2454] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da6034a055694fea8b554b4a98b35c92 in term 0.
I20260620 07:14:20.891330 25834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f39d3d5e3341bba73d3a3de692d2b5" candidate_uuid: "da6034a055694fea8b554b4a98b35c92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982427670b414556824d266b14d3a10d" is_pre_election: true
I20260620 07:14:20.891454 25902 leader_election.cc:302] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d017c0c3ce9740f09e8c731aa9a019f3, da6034a055694fea8b554b4a98b35c92; no voters:
I20260620 07:14:20.891849 26172 raft_consensus.cc:2781] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:14:20.891896 25834 raft_consensus.cc:2454] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da6034a055694fea8b554b4a98b35c92 in term 0.
I20260620 07:14:20.892028 26172 raft_consensus.cc:482] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:14:20.892160 26172 raft_consensus.cc:3037] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:20.894672 26172 raft_consensus.cc:504] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d017c0c3ce9740f09e8c731aa9a019f3" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34175 } } peers { permanent_uuid: "da6034a055694fea8b554b4a98b35c92" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45449 } } peers { permanent_uuid: "982427670b414556824d266b14d3a10d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38861 } }
I20260620 07:14:20.895268 26172 leader_election.cc:288] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [CANDIDATE]: Term 1 election: Requested vote from peers d017c0c3ce9740f09e8c731aa9a019f3 (127.6.53.67:34175), 982427670b414556824d266b14d3a10d (127.6.53.65:38861)
I20260620 07:14:20.895689 26101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f39d3d5e3341bba73d3a3de692d2b5" candidate_uuid: "da6034a055694fea8b554b4a98b35c92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d017c0c3ce9740f09e8c731aa9a019f3"
I20260620 07:14:20.895778 25834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f39d3d5e3341bba73d3a3de692d2b5" candidate_uuid: "da6034a055694fea8b554b4a98b35c92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982427670b414556824d266b14d3a10d"
I20260620 07:14:20.895875 26101 raft_consensus.cc:3037] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:20.896020 25834 raft_consensus.cc:3037] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:20.898149 26101 raft_consensus.cc:2454] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6034a055694fea8b554b4a98b35c92 in term 1.
I20260620 07:14:20.898149 25834 raft_consensus.cc:2454] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6034a055694fea8b554b4a98b35c92 in term 1.
I20260620 07:14:20.898514 25902 leader_election.cc:302] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d017c0c3ce9740f09e8c731aa9a019f3, da6034a055694fea8b554b4a98b35c92; no voters:
I20260620 07:14:20.898777 26172 raft_consensus.cc:2781] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:20.899199 26172 raft_consensus.cc:686] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [term 1 LEADER]: Becoming Leader. State: Replica: da6034a055694fea8b554b4a98b35c92, State: Running, Role: LEADER
I20260620 07:14:20.899735 26172 consensus_queue.cc:237] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d017c0c3ce9740f09e8c731aa9a019f3" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34175 } } peers { permanent_uuid: "da6034a055694fea8b554b4a98b35c92" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45449 } } peers { permanent_uuid: "982427670b414556824d266b14d3a10d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38861 } }
I20260620 07:14:20.904245 25689 catalog_manager.cc:5495] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 reported cstate change: term changed from 0 to 1, leader changed from <none> to da6034a055694fea8b554b4a98b35c92 (127.6.53.66). New cstate: current_term: 1 leader_uuid: "da6034a055694fea8b554b4a98b35c92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d017c0c3ce9740f09e8c731aa9a019f3" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6034a055694fea8b554b4a98b35c92" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "982427670b414556824d266b14d3a10d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38861 } health_report { overall_health: UNKNOWN } } }
I20260620 07:14:20.930454 26161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.925854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58112 (local address 127.6.53.66:45449)
0620 07:14:20.926058 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.926062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.926077 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:14:20.926150 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.926154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.926233 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.926366 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:14:20.926669 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.927367 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.928585 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.928964 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.929049 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.929453 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:14:20.929903 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:14:20.930039 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.930118 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.930176 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:14:20.937848 26101 raft_consensus.cc:1261] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3 [term 1 FOLLOWER]: Refusing update from remote peer da6034a055694fea8b554b4a98b35c92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:20.937885 25834 raft_consensus.cc:1261] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d [term 1 FOLLOWER]: Refusing update from remote peer da6034a055694fea8b554b4a98b35c92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:20.938851 26172 consensus_queue.cc:1035] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982427670b414556824d266b14d3a10d" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 38861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:20.939162 26175 consensus_queue.cc:1035] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d017c0c3ce9740f09e8c731aa9a019f3" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 34175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:20.953802 26179 mvcc.cc:205] Tried to move back new op lower bound from 7298824851192410112 to 7298824851049349120. Current Snapshot: MvccSnapshot[applied={T|T < 7298824851192410112}]
I20260620 07:14:20.965314 25690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:20.965616 25690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:20.969833 25690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c229a8ed98a4c43982583ffc0708ba6: Sending DeleteTablet for 3 replicas of tablet 64f39d3d5e3341bba73d3a3de692d2b5
I20260620 07:14:20.970924 26081 tablet_service.cc:1514] Processing DeleteTablet for tablet 64f39d3d5e3341bba73d3a3de692d2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:20 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:58196
I20260620 07:14:20.971231 25814 tablet_service.cc:1514] Processing DeleteTablet for tablet 64f39d3d5e3341bba73d3a3de692d2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:20 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:42840
I20260620 07:14:20.971258 25948 tablet_service.cc:1514] Processing DeleteTablet for tablet 64f39d3d5e3341bba73d3a3de692d2b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:20 UTC) from {username='kudu', principal='kudu/127.6.53.126@KRBTEST.COM'} at 127.0.0.1:58098
I20260620 07:14:20.973151 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 25751
I20260620 07:14:20.981459 26189 tablet_replica.cc:330] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92: stopping tablet replica
I20260620 07:14:20.981967 26189 raft_consensus.cc:2229] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:14:20.982713 26189 raft_consensus.cc:2258] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:20.985090 26189 ts_tablet_manager.cc:1905] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:20.990809 26188 tablet_replica.cc:330] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d: stopping tablet replica
I20260620 07:14:20.991287 26188 raft_consensus.cc:2229] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:20.991870 26188 raft_consensus.cc:2258] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:20.992671 26187 tablet_replica.cc:330] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3: stopping tablet replica
I20260620 07:14:20.993057 26187 raft_consensus.cc:2229] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:20.993559 26187 raft_consensus.cc:2258] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:20.995618 26188 ts_tablet_manager.cc:1905] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:20.996155 26187 ts_tablet_manager.cc:1905] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:21.003108 26188 ts_tablet_manager.cc:1918] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:21.003273 26188 log.cc:1196] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-0/wal/wals/64f39d3d5e3341bba73d3a3de692d2b5
I20260620 07:14:21.003816 26188 ts_tablet_manager.cc:1939] T 64f39d3d5e3341bba73d3a3de692d2b5 P 982427670b414556824d266b14d3a10d: Deleting consensus metadata
I20260620 07:14:21.005203 26189 ts_tablet_manager.cc:1918] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Jun 20 07:14:21 dist-test-slave-4n6h krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.65@KRBTEST.COM
I20260620 07:14:21.005375 26189 log.cc:1196] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/64f39d3d5e3341bba73d3a3de692d2b5
I20260620 07:14:21.005733 25675 catalog_manager.cc:4841] TS 982427670b414556824d266b14d3a10d (127.6.53.65:38861): tablet 64f39d3d5e3341bba73d3a3de692d2b5 (table test-table [id=5bd7604dfbe94ff698cdf68cb262c9ea]) successfully deleted
I20260620 07:14:21.005925 26189 ts_tablet_manager.cc:1939] T 64f39d3d5e3341bba73d3a3de692d2b5 P da6034a055694fea8b554b4a98b35c92: Deleting consensus metadata
I20260620 07:14:21.007817 26187 ts_tablet_manager.cc:1918] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:21.007990 26187 log.cc:1196] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/64f39d3d5e3341bba73d3a3de692d2b5
I20260620 07:14:21.008067 25675 catalog_manager.cc:4841] TS da6034a055694fea8b554b4a98b35c92 (127.6.53.66:45449): tablet 64f39d3d5e3341bba73d3a3de692d2b5 (table test-table [id=5bd7604dfbe94ff698cdf68cb262c9ea]) successfully deleted
I20260620 07:14:21.008474 26187 ts_tablet_manager.cc:1939] T 64f39d3d5e3341bba73d3a3de692d2b5 P d017c0c3ce9740f09e8c731aa9a019f3: Deleting consensus metadata
I20260620 07:14:21.010123 25675 catalog_manager.cc:4841] TS d017c0c3ce9740f09e8c731aa9a019f3 (127.6.53.67:34175): tablet 64f39d3d5e3341bba73d3a3de692d2b5 (table test-table [id=5bd7604dfbe94ff698cdf68cb262c9ea]) successfully deleted
I20260620 07:14:21.013419 26163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.974595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42856 (local address 127.6.53.65:38861)
0620 07:14:20.974813 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.974822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.994155 (+ 19333us) server_negotiation.cc:407] Connection header received
0620 07:14:20.994543 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.994549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.994656 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.994763 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:20.996829 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.997989 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.999896 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.000592 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.000696 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.006836 (+ 6140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:21.006849 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:21.006867 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:21.006909 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:21.010666 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:21.011349 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:21.011357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:21.011361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:21.011495 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:21.011885 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:21.011890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:21.011893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:21.012228 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:21.012530 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:21.012841 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 07:14:21.013068 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:14:21.014111 25874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:21.115279 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 25751
I20260620 07:14:21.141206 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 25885
Jun 20 07:14:21 dist-test-slave-4n6h krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.66@KRBTEST.COM
I20260620 07:14:21.157001 26161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.142107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.6.53.66:45449)
0620 07:14:21.142345 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:21.142353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:21.142374 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:14:21.142441 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:21.142444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:21.142515 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:21.142636 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:21.144065 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.144978 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.146192 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.146716 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.146848 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.151155 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:21.151167 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:21.151182 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:21.151213 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:21.154186 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:21.154899 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:21.154906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:21.154908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:21.155058 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:21.155444 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:21.155461 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:21.155463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:21.155804 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:21.156171 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:21.156404 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 07:14:21.156634 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:14:21.157876 26008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:21.269898 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 25885
I20260620 07:14:21.288363 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 26018
Jun 20 07:14:21 dist-test-slave-4n6h krb5kdc[25645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.67@KRBTEST.COM
I20260620 07:14:21.303761 26159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.289237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58204 (local address 127.6.53.67:34175)
0620 07:14:21.289415 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:21.289419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:21.289437 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:14:21.289562 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:21.289566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:21.289626 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:21.289709 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:21.291080 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.292048 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.293238 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.293605 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.293695 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.298478 (+ 4783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:21.298491 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:21.298506 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:21.298536 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:21.301235 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:21.301887 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:21.301894 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:21.301896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:21.302064 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:21.302513 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:21.302516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:21.302518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:21.302844 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:21.303097 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:21.303257 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 07:14:21.303405 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:14:21.304513 26141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:21.393306 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 26018
I20260620 07:14:21.410876 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 25658
I20260620 07:14:21.412045 25720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:21.521435 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 25658
2026-06-20T07:14:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4321 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T07:14:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:21Z Disabled control of system clock
I20260620 07:14:21.599169 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:39345
--webserver_interface=127.6.53.126
--webserver_port=0
--builtin_ntp_servers=127.6.53.84:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:39345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:14:21.941671 26198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:21.942107 26198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:21.942296 26198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:21.949680 26198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:21.949800 26198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:21.949885 26198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:21.949975 26198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:21.950058 26198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:21.959532 26198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:39345
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:39345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:21.960829 26198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:21.962613 26198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:21.975365 26204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:21.975605 26203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:21.975636 26206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:21.976845 26198 server_base.cc:1034] running on GCE node
I20260620 07:14:21.977689 26198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:21.979266 26198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:21.980504 26198 hybrid_clock.cc:648] HybridClock initialized: now 1781939661980441 us; error 63 us; skew 500 ppm
I20260620 07:14:21.980860 26198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:14:21.983340 26198 webserver.cc:460] Webserver started at http://127.6.53.126:43323/ using document root <none> and password file <none>
I20260620 07:14:21.984051 26198 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:21.984187 26198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:21.984519 26198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:21.987571 26198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "924ed7234f3f41518d628f1e29105eae"
format_stamp: "Formatted at 2026-06-20 07:14:21 on dist-test-slave-4n6h"
server_key: "cb381152009d737ad9dcec6df696dcc8"
server_key_iv: "1b05417a3e49322648cdedb8b68110f1"
server_key_version: "encryptionkey@0"
I20260620 07:14:21.988386 26198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "924ed7234f3f41518d628f1e29105eae"
format_stamp: "Formatted at 2026-06-20 07:14:21 on dist-test-slave-4n6h"
server_key: "cb381152009d737ad9dcec6df696dcc8"
server_key_iv: "1b05417a3e49322648cdedb8b68110f1"
server_key_version: "encryptionkey@0"
I20260620 07:14:21.993494 26198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:14:21.996915 26212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:21.998301 26198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:14:21.998512 26198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "924ed7234f3f41518d628f1e29105eae"
format_stamp: "Formatted at 2026-06-20 07:14:21 on dist-test-slave-4n6h"
server_key: "cb381152009d737ad9dcec6df696dcc8"
server_key_iv: "1b05417a3e49322648cdedb8b68110f1"
server_key_version: "encryptionkey@0"
I20260620 07:14:21.998741 26198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:22.034056 26198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:22.034998 26198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:22.035291 26198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:22.054286 26198 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:39345
I20260620 07:14:22.054348 26263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:39345 every 8 connection(s)
I20260620 07:14:22.055702 26198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:22.059463 26264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:22.064826 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 26198
I20260620 07:14:22.065025 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:22.065577 6357 external_mini_cluster.cc:1428] Setting key e1123b782ab75950f3f6c647dcbcf6e2
I20260620 07:14:22.071326 26264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae: Bootstrap starting.
I20260620 07:14:22.074404 26264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:22.075477 26264 log.cc:824] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:22.079115 26264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae: No bootstrap required, opened a new log
I20260620 07:14:22.079836 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:22.067666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.6.53.126:39345)
0620 07:14:22.068725 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:22.068736 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:22.068779 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:14:22.070567 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:22.070590 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:22.071027 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:22.071440 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:22.072004 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:22.073281 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:22.074706 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:22.075390 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:22.075595 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:22.076912 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:22.076923 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:22.076943 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:22.077669 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:22.077775 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:22.077806 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:22.077947 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:22.078196 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 07:14:22.078553 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":380,"threads_started":1}
I20260620 07:14:22.086900 26264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924ed7234f3f41518d628f1e29105eae" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39345 } }
I20260620 07:14:22.087294 26264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:22.087366 26264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924ed7234f3f41518d628f1e29105eae, State: Initialized, Role: FOLLOWER
I20260620 07:14:22.087878 26264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924ed7234f3f41518d628f1e29105eae" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39345 } }
I20260620 07:14:22.088117 26264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:22.088251 26264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:22.088447 26264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:22.091228 26264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924ed7234f3f41518d628f1e29105eae" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39345 } }
I20260620 07:14:22.091817 26264 leader_election.cc:302] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 924ed7234f3f41518d628f1e29105eae; no voters:
I20260620 07:14:22.092417 26264 leader_election.cc:288] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:22.092561 26269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:22.093312 26269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [term 1 LEADER]: Becoming Leader. State: Replica: 924ed7234f3f41518d628f1e29105eae, State: Running, Role: LEADER
I20260620 07:14:22.093830 26269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924ed7234f3f41518d628f1e29105eae" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39345 } }
I20260620 07:14:22.094692 26264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:22.100114 26271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 924ed7234f3f41518d628f1e29105eae. Latest consensus state: current_term: 1 leader_uuid: "924ed7234f3f41518d628f1e29105eae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924ed7234f3f41518d628f1e29105eae" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39345 } } }
I20260620 07:14:22.100476 26271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:22.101438 26279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:22.101369 26270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "924ed7234f3f41518d628f1e29105eae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924ed7234f3f41518d628f1e29105eae" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39345 } } }
I20260620 07:14:22.101692 26270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:22.106850 26279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:22.116322 26279 catalog_manager.cc:1347] Generated new cluster ID: 7b27f0a404a54580a2af4267dbfcf5f9
I20260620 07:14:22.116420 26279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:22.132755 26279 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:22.134043 26279 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:22.140899 26279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae: Generated new TSK 0
I20260620 07:14:22.141778 26279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:22.156199 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.65:0
--local_ip_for_outbound_sockets=127.6.53.65
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:39345
--builtin_ntp_servers=127.6.53.84:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:14:22.489087 26288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:22.489375 26288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:22.489509 26288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:22.497083 26288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:22.497187 26288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:22.497352 26288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.65
I20260620 07:14:22.506759 26288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.53.65
--webserver_port=0
--tserver_master_addrs=127.6.53.126:39345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.65
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:22.508119 26288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:22.509897 26288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:22.523313 26293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:22.523082 26296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:22.523255 26294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:22.524499 26288 server_base.cc:1034] running on GCE node
I20260620 07:14:22.525084 26288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:22.526010 26288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:22.527211 26288 hybrid_clock.cc:648] HybridClock initialized: now 1781939662527140 us; error 65 us; skew 500 ppm
I20260620 07:14:22.527540 26288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:14:22.529750 26288 webserver.cc:460] Webserver started at http://127.6.53.65:33395/ using document root <none> and password file <none>
I20260620 07:14:22.530496 26288 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:22.530606 26288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:22.530932 26288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:22.533763 26288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/data/instance:
uuid: "0a0e370bfa9442c6a5a81c9a282c8ef4"
format_stamp: "Formatted at 2026-06-20 07:14:22 on dist-test-slave-4n6h"
server_key: "258eb45cf143e61f38e2d928fc13d812"
server_key_iv: "fb8e49a640fd2b56ec184e1050d6c78c"
server_key_version: "encryptionkey@0"
I20260620 07:14:22.534595 26288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance:
uuid: "0a0e370bfa9442c6a5a81c9a282c8ef4"
format_stamp: "Formatted at 2026-06-20 07:14:22 on dist-test-slave-4n6h"
server_key: "258eb45cf143e61f38e2d928fc13d812"
server_key_iv: "fb8e49a640fd2b56ec184e1050d6c78c"
server_key_version: "encryptionkey@0"
I20260620 07:14:22.539535 26288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 07:14:22.542903 26302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:22.544312 26288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:14:22.544510 26288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/wal
uuid: "0a0e370bfa9442c6a5a81c9a282c8ef4"
format_stamp: "Formatted at 2026-06-20 07:14:22 on dist-test-slave-4n6h"
server_key: "258eb45cf143e61f38e2d928fc13d812"
server_key_iv: "fb8e49a640fd2b56ec184e1050d6c78c"
server_key_version: "encryptionkey@0"
I20260620 07:14:22.544744 26288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:22.563469 26288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:22.564320 26288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:22.564605 26288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:22.565544 26288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:22.567018 26288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:22.567154 26288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:22.567283 26288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:22.567369 26288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:22.600349 26288 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.65:37849
I20260620 07:14:22.600485 26414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.65:37849 every 8 connection(s)
I20260620 07:14:22.601761 26288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/data/info.pb
I20260620 07:14:22.604108 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 26288
I20260620 07:14:22.604283 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-0/wal/instance
I20260620 07:14:22.604848 6357 external_mini_cluster.cc:1428] Setting key 0fa49e76db69cc3512c8f302d639f238
I20260620 07:14:22.607963 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.66:0
--local_ip_for_outbound_sockets=127.6.53.66
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:39345
--builtin_ntp_servers=127.6.53.84:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:14:22.616060 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:22.604386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.65:56419 (local address 127.6.53.126:39345)
0620 07:14:22.604838 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:22.604845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:22.606155 (+ 1310us) server_negotiation.cc:407] Connection header received
0620 07:14:22.608615 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:22.608620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:22.608678 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:22.608758 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:22.610121 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:22.610876 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:22.613254 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:22.613665 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:22.613758 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:22.615317 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:22.615322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:22.615325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:22.615342 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:22.615381 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:22.615392 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:22.615465 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:22.615717 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 07:14:22.615790 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":216,"threads_started":1}
I20260620 07:14:22.617293 26418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:22.604994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:39345 (local address 127.6.53.65:56419)
0620 07:14:22.606012 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:22.606052 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:22.608331 (+ 2279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:22.609013 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:22.609027 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:22.609144 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:22.609920 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:22.609946 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:22.611042 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:22.611050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:22.613106 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:22.613124 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:22.613881 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:22.613890 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:22.614198 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:22.614970 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:14:22.614975 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:22.615029 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:14:22.615133 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:22.615489 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:22.615502 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:22.615514 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 07:14:22.615701 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 07:14:22.616115 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":328,"threads_started":1}
I20260620 07:14:22.618840 26415 heartbeater.cc:347] Connected to a master server at 127.6.53.126:39345
I20260620 07:14:22.619133 26415 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:22.619814 26415 heartbeater.cc:511] Master 127.6.53.126:39345 requested a full tablet report, sending...
I20260620 07:14:22.621935 26229 ts_manager.cc:194] Registered new tserver with Master: 0a0e370bfa9442c6a5a81c9a282c8ef4 (127.6.53.65:37849)
I20260620 07:14:22.624138 26229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.53.65:56419
W20260620 07:14:22.937493 26419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:22.937762 26419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:22.937855 26419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:22.944934 26419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:22.945032 26419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:22.945138 26419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.66
I20260620 07:14:22.954357 26419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.53.66
--webserver_port=0
--tserver_master_addrs=127.6.53.126:39345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.66
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:22.955524 26419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:22.957190 26419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:22.969719 26424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:22.969796 26427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:22.969679 26425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:22.970680 26419 server_base.cc:1034] running on GCE node
I20260620 07:14:22.971248 26419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:22.972121 26419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:22.973347 26419 hybrid_clock.cc:648] HybridClock initialized: now 1781939662973300 us; error 42 us; skew 500 ppm
I20260620 07:14:22.973655 26419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:14:22.975942 26419 webserver.cc:460] Webserver started at http://127.6.53.66:33557/ using document root <none> and password file <none>
I20260620 07:14:22.976608 26419 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:22.976734 26419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:22.977053 26419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:22.979807 26419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/data/instance:
uuid: "84f74149373f4e7c9abd04d5004bb24b"
format_stamp: "Formatted at 2026-06-20 07:14:22 on dist-test-slave-4n6h"
server_key: "c386f4108e606d1a4fe293f99b4b4936"
server_key_iv: "a89b545eaf91bf15124870698f2ecdaf"
server_key_version: "encryptionkey@0"
I20260620 07:14:22.980572 26419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance:
uuid: "84f74149373f4e7c9abd04d5004bb24b"
format_stamp: "Formatted at 2026-06-20 07:14:22 on dist-test-slave-4n6h"
server_key: "c386f4108e606d1a4fe293f99b4b4936"
server_key_iv: "a89b545eaf91bf15124870698f2ecdaf"
server_key_version: "encryptionkey@0"
I20260620 07:14:22.985360 26419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 07:14:22.988610 26433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:22.989802 26419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:14:22.989964 26419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/wal
uuid: "84f74149373f4e7c9abd04d5004bb24b"
format_stamp: "Formatted at 2026-06-20 07:14:22 on dist-test-slave-4n6h"
server_key: "c386f4108e606d1a4fe293f99b4b4936"
server_key_iv: "a89b545eaf91bf15124870698f2ecdaf"
server_key_version: "encryptionkey@0"
I20260620 07:14:22.990180 26419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:23.020027 26419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:23.020872 26419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:23.021157 26419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:23.022140 26419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:23.023591 26419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:23.023725 26419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:23.023846 26419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:23.023933 26419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:23.056324 26419 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.66:45121
I20260620 07:14:23.056449 26545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.66:45121 every 8 connection(s)
I20260620 07:14:23.057525 26419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/data/info.pb
I20260620 07:14:23.065640 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 26419
I20260620 07:14:23.065821 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/wal/instance
I20260620 07:14:23.066316 6357 external_mini_cluster.cc:1428] Setting key e9acde3aa44a473065c8b9d3b161631c
I20260620 07:14:23.069133 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.53.67:0
--local_ip_for_outbound_sockets=127.6.53.67
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:39345
--builtin_ntp_servers=127.6.53.84:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:14:23.071980 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.060276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:59083 (local address 127.6.53.126:39345)
0620 07:14:23.060481 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:23.060488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:23.061909 (+ 1421us) server_negotiation.cc:407] Connection header received
0620 07:14:23.064324 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:23.064330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:23.064401 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:23.064485 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:23.065804 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.066967 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.068755 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.069383 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.069506 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.071181 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:23.071188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:23.071193 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:23.071211 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:23.071247 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:23.071261 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:23.071346 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:23.071597 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 07:14:23.071685 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:14:23.073393 26548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.060755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:39345 (local address 127.6.53.66:59083)
0620 07:14:23.061750 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:23.061795 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:23.064024 (+ 2229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:23.064742 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:23.064753 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:23.064859 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:23.065606 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.065628 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.067127 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.067134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.068610 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.068624 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.069608 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.069617 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.069978 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.070851 (+ 873us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:14:23.070857 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:23.070903 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:14:23.070992 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:23.071364 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:23.071376 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:23.071386 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 07:14:23.071558 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 07:14:23.071925 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":287,"threads_started":1}
I20260620 07:14:23.074736 26546 heartbeater.cc:347] Connected to a master server at 127.6.53.126:39345
I20260620 07:14:23.075039 26546 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:23.075726 26546 heartbeater.cc:511] Master 127.6.53.126:39345 requested a full tablet report, sending...
I20260620 07:14:23.077046 26229 ts_manager.cc:194] Registered new tserver with Master: 84f74149373f4e7c9abd04d5004bb24b (127.6.53.66:45121)
I20260620 07:14:23.078111 26229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.53.66:59083
W20260620 07:14:23.416536 26549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:23.416791 26549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:23.416942 26549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:23.424621 26549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:23.424710 26549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:23.424901 26549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.53.67
I20260620 07:14:23.433652 26549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.53.67
--webserver_port=0
--tserver_master_addrs=127.6.53.126:39345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.53.67
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:23.434943 26549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:23.436555 26549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:23.449235 26557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:23.449328 26555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:23.449328 26554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:23.450649 26549 server_base.cc:1034] running on GCE node
I20260620 07:14:23.451232 26549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:23.452122 26549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:23.453315 26549 hybrid_clock.cc:648] HybridClock initialized: now 1781939663453258 us; error 52 us; skew 500 ppm
I20260620 07:14:23.453615 26549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:14:23.455760 26549 webserver.cc:460] Webserver started at http://127.6.53.67:37389/ using document root <none> and password file <none>
I20260620 07:14:23.456416 26549 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:23.456526 26549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:23.456852 26549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:23.459626 26549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/data/instance:
uuid: "42d641df6bd847e088faa1ab9c0f1064"
format_stamp: "Formatted at 2026-06-20 07:14:23 on dist-test-slave-4n6h"
server_key: "5dc2697b5cfa8888b46f61ee29a119ed"
server_key_iv: "7dd26fd8ebf13715e78a3297f03bebc4"
server_key_version: "encryptionkey@0"
I20260620 07:14:23.460366 26549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance:
uuid: "42d641df6bd847e088faa1ab9c0f1064"
format_stamp: "Formatted at 2026-06-20 07:14:23 on dist-test-slave-4n6h"
server_key: "5dc2697b5cfa8888b46f61ee29a119ed"
server_key_iv: "7dd26fd8ebf13715e78a3297f03bebc4"
server_key_version: "encryptionkey@0"
I20260620 07:14:23.465142 26549 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 07:14:23.468355 26563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:23.469534 26549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 07:14:23.469694 26549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/wal
uuid: "42d641df6bd847e088faa1ab9c0f1064"
format_stamp: "Formatted at 2026-06-20 07:14:23 on dist-test-slave-4n6h"
server_key: "5dc2697b5cfa8888b46f61ee29a119ed"
server_key_iv: "7dd26fd8ebf13715e78a3297f03bebc4"
server_key_version: "encryptionkey@0"
I20260620 07:14:23.469897 26549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:23.484447 26549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:23.485165 26549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:23.485422 26549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:23.486352 26549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:23.487694 26549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:23.487805 26549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:23.487941 26549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:23.488032 26549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:23.521255 26549 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.67:45291
I20260620 07:14:23.521341 26675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.67:45291 every 8 connection(s)
I20260620 07:14:23.522608 26549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/data/info.pb
I20260620 07:14:23.527491 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 26549
I20260620 07:14:23.527666 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/wal/instance
I20260620 07:14:23.528218 6357 external_mini_cluster.cc:1428] Setting key 77e8435176d0a2a29e454bc4038b33c7
I20260620 07:14:23.535931 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.525085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.67:56733 (local address 127.6.53.126:39345)
0620 07:14:23.525263 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:23.525268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:23.526531 (+ 1263us) server_negotiation.cc:407] Connection header received
0620 07:14:23.528844 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:23.528850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:23.528940 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:23.529055 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:23.530161 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.531155 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.533189 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.533606 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.533742 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.535150 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:23.535155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:23.535158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:23.535175 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:23.535205 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:23.535217 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:23.535314 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:23.535529 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 07:14:23.535621 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:14:23.536846 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.525519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:39345 (local address 127.6.53.67:56733)
0620 07:14:23.526382 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:23.526427 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:23.528563 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:23.529272 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:23.529279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:23.529357 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:23.530004 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.530021 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.531291 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.531303 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.533036 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.533049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.533825 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.533832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.534081 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.534831 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:14:23.534834 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:23.534869 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:14:23.534940 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:23.535313 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:23.535325 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:23.535335 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 07:14:23.535519 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 07:14:23.535876 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":257,"threads_started":1}
I20260620 07:14:23.538041 26676 heartbeater.cc:347] Connected to a master server at 127.6.53.126:39345
I20260620 07:14:23.538335 26676 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:23.538964 26676 heartbeater.cc:511] Master 127.6.53.126:39345 requested a full tablet report, sending...
I20260620 07:14:23.540162 26229 ts_manager.cc:194] Registered new tserver with Master: 42d641df6bd847e088faa1ab9c0f1064 (127.6.53.67:45291)
I20260620 07:14:23.540908 26229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.53.67:56733
I20260620 07:14:23.543071 6357 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:14:23.553190 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.547320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.6.53.126:39345)
0620 07:14:23.547557 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:23.547562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:23.547996 (+ 434us) server_negotiation.cc:407] Connection header received
0620 07:14:23.548164 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:23.548168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:23.548272 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:23.548385 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:23.548866 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.549602 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.551094 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.551482 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.551579 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.552648 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:23.552652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:23.552654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:23.552670 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:23.552694 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:23.552704 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:23.552768 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:23.552892 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 07:14:23.552956 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 07:14:23.556782 26229 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:14:23.563711 26229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:14:23.601971 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.575436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:45291 (local address 127.0.0.1:51382)
0620 07:14:23.590797 (+ 15361us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:23.590815 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:23.590929 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:23.592087 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:23.592099 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:23.592141 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:23.592612 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.592628 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.595218 (+ 2590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.595226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.599261 (+ 4035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.599276 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.600978 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.600986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.601192 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.601246 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 07:14:23.601383 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 07:14:23.601599 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15204,"mutex_wait_us":1,"thread_start_us":14104,"threads_started":1}
I20260620 07:14:23.602016 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.575626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.6.53.67:45291)
0620 07:14:23.589128 (+ 13502us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:23.589142 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:23.591063 (+ 1921us) server_negotiation.cc:407] Connection header received
0620 07:14:23.591179 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:23.591191 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:23.591495 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:23.591706 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:23.592893 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.595010 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.599467 (+ 4457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.600770 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.600913 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.601186 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:23.601422 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 07:14:23.601599 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13262,"thread_start_us":13535,"threads_started":1}
I20260620 07:14:23.606293 26689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.575044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37849 (local address 127.0.0.1:53854)
0620 07:14:23.590016 (+ 14972us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:23.590067 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:23.590291 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:23.593047 (+ 2756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:23.593055 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:23.593089 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:23.593705 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.593722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.596377 (+ 2655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.596386 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.603299 (+ 6913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.603313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.605516 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.605522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.605696 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.605729 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:14:23.605816 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 07:14:23.605935 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14659,"mutex_wait_us":1,"thread_start_us":15449,"threads_started":1}
I20260620 07:14:23.606519 26611 tablet_service.cc:1467] Processing CreateTablet for tablet 50acdb51e14f4f30bf37ec0324381d6f (DEFAULT_TABLE table=test-table [id=40e1773a8b75413a92ae5d613d415541]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:23.607280 26690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.575546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53854 (local address 127.6.53.65:37849)
0620 07:14:23.590962 (+ 15416us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:23.590970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:23.591002 (+ 32us) server_negotiation.cc:407] Connection header received
0620 07:14:23.592222 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:23.592235 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:23.592518 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:23.592729 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:23.593936 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.596206 (+ 2270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.603570 (+ 7364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.605006 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.605139 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.605394 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:23.606741 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0620 07:14:23.606904 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15247,"thread_start_us":15113,"threads_started":1}
I20260620 07:14:23.608001 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.591018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.66:45121 (local address 127.0.0.1:33192)
0620 07:14:23.599382 (+ 8364us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:23.599400 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:23.599513 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:23.600136 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:23.600141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:23.600166 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:23.600529 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.600540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.602491 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.602496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.605204 (+ 2708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.605217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.607248 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.607254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.607428 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.607464 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:14:23.607584 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 07:14:23.607703 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8232,"thread_start_us":8082,"threads_started":1}
I20260620 07:14:23.608124 26693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.591123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.6.53.66:45121)
0620 07:14:23.597659 (+ 6536us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:23.597667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:23.599627 (+ 1960us) server_negotiation.cc:407] Connection header received
0620 07:14:23.599730 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:23.599737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:23.599951 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:23.600132 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:23.600709 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.602325 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.605399 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.606808 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.606942 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.607130 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:23.607633 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 07:14:23.607777 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6390,"thread_start_us":6284,"threads_started":1}
I20260620 07:14:23.608495 26611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50acdb51e14f4f30bf37ec0324381d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:23.610294 26350 tablet_service.cc:1467] Processing CreateTablet for tablet 50acdb51e14f4f30bf37ec0324381d6f (DEFAULT_TABLE table=test-table [id=40e1773a8b75413a92ae5d613d415541]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:23.611552 26350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50acdb51e14f4f30bf37ec0324381d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:23.613430 26481 tablet_service.cc:1467] Processing CreateTablet for tablet 50acdb51e14f4f30bf37ec0324381d6f (DEFAULT_TABLE table=test-table [id=40e1773a8b75413a92ae5d613d415541]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:23.614671 26481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50acdb51e14f4f30bf37ec0324381d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:23.621779 26694 tablet_bootstrap.cc:492] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4: Bootstrap starting.
I20260620 07:14:23.623673 26695 tablet_bootstrap.cc:492] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064: Bootstrap starting.
I20260620 07:14:23.626607 26694 tablet_bootstrap.cc:654] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:23.627079 26695 tablet_bootstrap.cc:654] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:23.627158 26415 heartbeater.cc:503] Master 127.6.53.126:39345 was elected leader, sending a full tablet report...
I20260620 07:14:23.627982 26694 log.cc:824] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:23.628237 26695 log.cc:824] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:23.629956 26696 tablet_bootstrap.cc:492] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b: Bootstrap starting.
I20260620 07:14:23.632823 26695 tablet_bootstrap.cc:492] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064: No bootstrap required, opened a new log
I20260620 07:14:23.633087 26695 ts_tablet_manager.cc:1397] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 07:14:23.633620 26696 tablet_bootstrap.cc:654] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:23.634022 26694 tablet_bootstrap.cc:492] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4: No bootstrap required, opened a new log
I20260620 07:14:23.634289 26694 ts_tablet_manager.cc:1397] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20260620 07:14:23.634801 26696 log.cc:824] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:23.637493 26696 tablet_bootstrap.cc:492] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b: No bootstrap required, opened a new log
I20260620 07:14:23.637729 26696 ts_tablet_manager.cc:1397] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 07:14:23.641607 26694 raft_consensus.cc:348] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0e370bfa9442c6a5a81c9a282c8ef4" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37849 } } peers { permanent_uuid: "42d641df6bd847e088faa1ab9c0f1064" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 45291 } } peers { permanent_uuid: "84f74149373f4e7c9abd04d5004bb24b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45121 } }
I20260620 07:14:23.641939 26694 raft_consensus.cc:374] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:23.642061 26694 raft_consensus.cc:729] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a0e370bfa9442c6a5a81c9a282c8ef4, State: Initialized, Role: FOLLOWER
I20260620 07:14:23.642668 26694 consensus_queue.cc:260] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0e370bfa9442c6a5a81c9a282c8ef4" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37849 } } peers { permanent_uuid: "42d641df6bd847e088faa1ab9c0f1064" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 45291 } } peers { permanent_uuid: "84f74149373f4e7c9abd04d5004bb24b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45121 } }
I20260620 07:14:23.642788 26695 raft_consensus.cc:348] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0e370bfa9442c6a5a81c9a282c8ef4" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37849 } } peers { permanent_uuid: "42d641df6bd847e088faa1ab9c0f1064" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 45291 } } peers { permanent_uuid: "84f74149373f4e7c9abd04d5004bb24b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45121 } }
I20260620 07:14:23.643252 26695 raft_consensus.cc:374] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:23.643374 26695 raft_consensus.cc:729] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d641df6bd847e088faa1ab9c0f1064, State: Initialized, Role: FOLLOWER
I20260620 07:14:23.643996 26695 consensus_queue.cc:260] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0e370bfa9442c6a5a81c9a282c8ef4" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37849 } } peers { permanent_uuid: "42d641df6bd847e088faa1ab9c0f1064" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 45291 } } peers { permanent_uuid: "84f74149373f4e7c9abd04d5004bb24b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45121 } }
I20260620 07:14:23.645112 26694 ts_tablet_manager.cc:1428] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260620 07:14:23.649523 26696 raft_consensus.cc:348] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0e370bfa9442c6a5a81c9a282c8ef4" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37849 } } peers { permanent_uuid: "42d641df6bd847e088faa1ab9c0f1064" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 45291 } } peers { permanent_uuid: "84f74149373f4e7c9abd04d5004bb24b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45121 } }
I20260620 07:14:23.650108 26696 raft_consensus.cc:374] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:23.650225 26696 raft_consensus.cc:729] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f74149373f4e7c9abd04d5004bb24b, State: Initialized, Role: FOLLOWER
I20260620 07:14:23.650960 26676 heartbeater.cc:503] Master 127.6.53.126:39345 was elected leader, sending a full tablet report...
I20260620 07:14:23.650880 26696 consensus_queue.cc:260] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0e370bfa9442c6a5a81c9a282c8ef4" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37849 } } peers { permanent_uuid: "42d641df6bd847e088faa1ab9c0f1064" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 45291 } } peers { permanent_uuid: "84f74149373f4e7c9abd04d5004bb24b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45121 } }
I20260620 07:14:23.651712 26695 ts_tablet_manager.cc:1428] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064: Time spent starting tablet: real 0.019s user 0.013s sys 0.006s
I20260620 07:14:23.651966 26546 heartbeater.cc:503] Master 127.6.53.126:39345 was elected leader, sending a full tablet report...
I20260620 07:14:23.652508 26696 ts_tablet_manager.cc:1428] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260620 07:14:23.666662 26702 raft_consensus.cc:482] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:23.666908 26702 raft_consensus.cc:504] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0e370bfa9442c6a5a81c9a282c8ef4" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37849 } } peers { permanent_uuid: "42d641df6bd847e088faa1ab9c0f1064" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 45291 } } peers { permanent_uuid: "84f74149373f4e7c9abd04d5004bb24b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45121 } }
I20260620 07:14:23.668355 26702 leader_election.cc:288] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a0e370bfa9442c6a5a81c9a282c8ef4 (127.6.53.65:37849), 42d641df6bd847e088faa1ab9c0f1064 (127.6.53.67:45291)
I20260620 07:14:23.676319 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.668839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:55983 (local address 127.6.53.67:45291)
0620 07:14:23.669021 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:23.669027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:23.669794 (+ 767us) server_negotiation.cc:407] Connection header received
0620 07:14:23.670003 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:23.670008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:23.670083 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:23.670182 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:23.671022 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.672127 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.674647 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.675415 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.675560 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.675666 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:23.675946 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 07:14:23.676078 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:14:23.676360 26690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.668839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.53.66:54211 (local address 127.6.53.65:37849)
0620 07:14:23.669021 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:23.669028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:23.669795 (+ 767us) server_negotiation.cc:407] Connection header received
0620 07:14:23.670003 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:23.670008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:23.670088 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:23.670182 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:23.671013 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.672095 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.674716 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.675464 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.675595 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.675699 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:23.675995 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 07:14:23.676131 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:14:23.676546 26703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.668798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.67:45291 (local address 127.6.53.66:55983)
0620 07:14:23.669663 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:23.669694 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:23.669854 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:23.670209 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:23.670214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:23.670270 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:23.670854 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.670864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.672264 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.672270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.674459 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.674472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.675617 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.675623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.675779 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.675819 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 07:14:23.675920 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 07:14:23.676046 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":188,"spinlock_wait_cycles":21632,"thread_start_us":437,"threads_started":1}
I20260620 07:14:23.676566 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.668798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37849 (local address 127.6.53.66:54211)
0620 07:14:23.669684 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:23.669698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:23.669854 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:23.670209 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:23.670214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:23.670276 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:23.670788 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.670806 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.672293 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.672297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.674504 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.674520 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.675656 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.675662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.675840 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.675862 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:14:23.675968 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 07:14:23.676099 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":325,"spinlock_wait_cycles":22272,"thread_start_us":419,"threads_started":1}
I20260620 07:14:23.677531 26631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50acdb51e14f4f30bf37ec0324381d6f" candidate_uuid: "84f74149373f4e7c9abd04d5004bb24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d641df6bd847e088faa1ab9c0f1064" is_pre_election: true
I20260620 07:14:23.677619 26370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50acdb51e14f4f30bf37ec0324381d6f" candidate_uuid: "84f74149373f4e7c9abd04d5004bb24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a0e370bfa9442c6a5a81c9a282c8ef4" is_pre_election: true
I20260620 07:14:23.678020 26631 raft_consensus.cc:2454] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84f74149373f4e7c9abd04d5004bb24b in term 0.
I20260620 07:14:23.678119 26370 raft_consensus.cc:2454] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84f74149373f4e7c9abd04d5004bb24b in term 0.
I20260620 07:14:23.678764 26434 leader_election.cc:302] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a0e370bfa9442c6a5a81c9a282c8ef4, 42d641df6bd847e088faa1ab9c0f1064, 84f74149373f4e7c9abd04d5004bb24b; no voters:
I20260620 07:14:23.679137 26702 raft_consensus.cc:2781] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:14:23.679275 26702 raft_consensus.cc:482] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:14:23.679388 26702 raft_consensus.cc:3037] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:23.681479 26702 raft_consensus.cc:504] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0e370bfa9442c6a5a81c9a282c8ef4" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37849 } } peers { permanent_uuid: "42d641df6bd847e088faa1ab9c0f1064" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 45291 } } peers { permanent_uuid: "84f74149373f4e7c9abd04d5004bb24b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45121 } }
I20260620 07:14:23.682080 26702 leader_election.cc:288] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [CANDIDATE]: Term 1 election: Requested vote from peers 0a0e370bfa9442c6a5a81c9a282c8ef4 (127.6.53.65:37849), 42d641df6bd847e088faa1ab9c0f1064 (127.6.53.67:45291)
I20260620 07:14:23.682479 26370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50acdb51e14f4f30bf37ec0324381d6f" candidate_uuid: "84f74149373f4e7c9abd04d5004bb24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a0e370bfa9442c6a5a81c9a282c8ef4"
I20260620 07:14:23.682574 26631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50acdb51e14f4f30bf37ec0324381d6f" candidate_uuid: "84f74149373f4e7c9abd04d5004bb24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d641df6bd847e088faa1ab9c0f1064"
I20260620 07:14:23.682684 26370 raft_consensus.cc:3037] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:23.682768 26631 raft_consensus.cc:3037] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:23.685834 26631 raft_consensus.cc:2454] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84f74149373f4e7c9abd04d5004bb24b in term 1.
I20260620 07:14:23.685834 26370 raft_consensus.cc:2454] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84f74149373f4e7c9abd04d5004bb24b in term 1.
I20260620 07:14:23.686233 26436 leader_election.cc:302] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a0e370bfa9442c6a5a81c9a282c8ef4, 42d641df6bd847e088faa1ab9c0f1064, 84f74149373f4e7c9abd04d5004bb24b; no voters:
I20260620 07:14:23.686554 26702 raft_consensus.cc:2781] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:23.686949 26702 raft_consensus.cc:686] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [term 1 LEADER]: Becoming Leader. State: Replica: 84f74149373f4e7c9abd04d5004bb24b, State: Running, Role: LEADER
I20260620 07:14:23.687386 26702 consensus_queue.cc:237] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0e370bfa9442c6a5a81c9a282c8ef4" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37849 } } peers { permanent_uuid: "42d641df6bd847e088faa1ab9c0f1064" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 45291 } } peers { permanent_uuid: "84f74149373f4e7c9abd04d5004bb24b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45121 } }
I20260620 07:14:23.691870 26229 catalog_manager.cc:5495] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b reported cstate change: term changed from 0 to 1, leader changed from <none> to 84f74149373f4e7c9abd04d5004bb24b (127.6.53.66). New cstate: current_term: 1 leader_uuid: "84f74149373f4e7c9abd04d5004bb24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0e370bfa9442c6a5a81c9a282c8ef4" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42d641df6bd847e088faa1ab9c0f1064" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 45291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84f74149373f4e7c9abd04d5004bb24b" member_type: VOTER last_known_addr { host: "127.6.53.66" port: 45121 } health_report { overall_health: HEALTHY } } }
I20260620 07:14:23.707445 26693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.702725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.6.53.66:45121)
0620 07:14:23.702887 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:23.702891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:23.703025 (+ 134us) server_negotiation.cc:407] Connection header received
0620 07:14:23.703182 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:23.703185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:23.703269 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:23.703411 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:14:23.703743 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.704391 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.705639 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.706020 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.706112 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.706556 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:14:23.707022 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:14:23.707126 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:23.707200 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 07:14:23.707257 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:14:23.715102 26631 raft_consensus.cc:1261] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064 [term 1 FOLLOWER]: Refusing update from remote peer 84f74149373f4e7c9abd04d5004bb24b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:23.715102 26370 raft_consensus.cc:1261] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4 [term 1 FOLLOWER]: Refusing update from remote peer 84f74149373f4e7c9abd04d5004bb24b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:23.716055 26705 consensus_queue.cc:1035] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42d641df6bd847e088faa1ab9c0f1064" member_type: VOTER last_known_addr { host: "127.6.53.67" port: 45291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:23.716430 26702 consensus_queue.cc:1035] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a0e370bfa9442c6a5a81c9a282c8ef4" member_type: VOTER last_known_addr { host: "127.6.53.65" port: 37849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:23.727309 26708 mvcc.cc:205] Tried to move back new op lower bound from 7298824862567333888 to 7298824862467375104. Current Snapshot: MvccSnapshot[applied={T|T < 7298824862567333888}]
I20260620 07:14:23.738811 26228 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:23.739054 26228 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:23.742375 26228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 924ed7234f3f41518d628f1e29105eae: Sending DeleteTablet for 3 replicas of tablet 50acdb51e14f4f30bf37ec0324381d6f
I20260620 07:14:23.743388 26350 tablet_service.cc:1514] Processing DeleteTablet for tablet 50acdb51e14f4f30bf37ec0324381d6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:23 UTC) from {username='slave'} at 127.0.0.1:53854
I20260620 07:14:23.743476 26611 tablet_service.cc:1514] Processing DeleteTablet for tablet 50acdb51e14f4f30bf37ec0324381d6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:23 UTC) from {username='slave'} at 127.0.0.1:51382
I20260620 07:14:23.743687 26481 tablet_service.cc:1514] Processing DeleteTablet for tablet 50acdb51e14f4f30bf37ec0324381d6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:23 UTC) from {username='slave'} at 127.0.0.1:33192
I20260620 07:14:23.744680 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 26288
I20260620 07:14:23.763342 26718 tablet_replica.cc:330] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4: stopping tablet replica
I20260620 07:14:23.763980 26718 raft_consensus.cc:2229] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:23.764616 26718 raft_consensus.cc:2258] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:23.764745 26720 tablet_replica.cc:330] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b: stopping tablet replica
I20260620 07:14:23.765584 26720 raft_consensus.cc:2229] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:14:23.767452 26718 ts_tablet_manager.cc:1905] T 50acdb51e14f4f30bf37ec0324381d6f P 0a0e370bfa9442c6a5a81c9a282c8ef4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:23.767817 26690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.745834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.6.53.65:37849)
0620 07:14:23.746064 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:23.746073 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:23.756604 (+ 10531us) server_negotiation.cc:407] Connection header received
0620 07:14:23.757162 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:23.757172 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:23.757314 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:23.757426 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:23.758285 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.759769 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.762291 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.763308 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.763434 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.766148 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:23.766159 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:23.766176 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:23.766937 (+ 761us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:23.766991 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:23.767014 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:23.767115 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:23.767357 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 07:14:23.767472 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:14:23.769232 26410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:23.769836 26719 tablet_replica.cc:330] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064: stopping tablet replica
I20260620 07:14:23.770274 26719 raft_consensus.cc:2229] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:23.770632 26720 raft_consensus.cc:2258] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:23.770766 26719 raft_consensus.cc:2258] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:23.772779 26719 ts_tablet_manager.cc:1905] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:23.772825 26720 ts_tablet_manager.cc:1905] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:23.779929 26719 ts_tablet_manager.cc:1918] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:23.780049 26720 ts_tablet_manager.cc:1918] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:23.780170 26719 log.cc:1196] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-2/wal/wals/50acdb51e14f4f30bf37ec0324381d6f
I20260620 07:14:23.780236 26720 log.cc:1196] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b: Deleting WAL directory at /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939352012798-6357-0/minicluster-data/ts-1/wal/wals/50acdb51e14f4f30bf37ec0324381d6f
I20260620 07:14:23.780807 26720 ts_tablet_manager.cc:1939] T 50acdb51e14f4f30bf37ec0324381d6f P 84f74149373f4e7c9abd04d5004bb24b: Deleting consensus metadata
I20260620 07:14:23.780807 26719 ts_tablet_manager.cc:1939] T 50acdb51e14f4f30bf37ec0324381d6f P 42d641df6bd847e088faa1ab9c0f1064: Deleting consensus metadata
I20260620 07:14:23.782557 26213 catalog_manager.cc:4841] TS 42d641df6bd847e088faa1ab9c0f1064 (127.6.53.67:45291): tablet 50acdb51e14f4f30bf37ec0324381d6f (table test-table [id=40e1773a8b75413a92ae5d613d415541]) successfully deleted
I20260620 07:14:23.782678 26215 catalog_manager.cc:4841] TS 84f74149373f4e7c9abd04d5004bb24b (127.6.53.66:45121): tablet 50acdb51e14f4f30bf37ec0324381d6f (table test-table [id=40e1773a8b75413a92ae5d613d415541]) successfully deleted
I20260620 07:14:23.891757 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 26288
I20260620 07:14:23.909627 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 26419
W20260620 07:14:23.909672 26215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.53.65:37849 (error 108)
I20260620 07:14:23.911955 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.911421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37849 (local address 127.0.0.1:53872)
0620 07:14:23.911661 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:23.911788 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:37849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260620 07:14:23.912281 26215 catalog_manager.cc:4584] TS 0a0e370bfa9442c6a5a81c9a282c8ef4 (127.6.53.65:37849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50acdb51e14f4f30bf37ec0324381d6f: Network error: Client connection negotiation failed: client connection to 127.6.53.65:37849: connect: Connection refused (error 111)
I20260620 07:14:23.921932 26693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.910533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.6.53.66:45121)
0620 07:14:23.910722 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:23.910729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:23.911157 (+ 428us) server_negotiation.cc:407] Connection header received
0620 07:14:23.911339 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:23.911344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:23.911483 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:23.911592 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:23.912260 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.913164 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.915093 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.915528 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.915625 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.920402 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:23.920411 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:23.920424 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:23.921086 (+ 662us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:23.921133 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:23.921154 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:23.921239 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:23.921415 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 07:14:23.921500 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:14:23.922747 26541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:23.965839 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.965279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37849 (local address 127.0.0.1:53880)
0620 07:14:23.965528 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:23.965647 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:37849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 07:14:23.966981 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.966593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37849 (local address 127.0.0.1:53890)
0620 07:14:23.966743 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:23.966824 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:37849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 07:14:24.027575 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:24.027014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37849 (local address 127.0.0.1:53900)
0620 07:14:24.027252 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:24.027363 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:37849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 07:14:24.028630 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:24.028240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37849 (local address 127.0.0.1:53912)
0620 07:14:24.028415 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:24.028496 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:37849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 07:14:24.032689 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 26419
I20260620 07:14:24.050948 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 26549
I20260620 07:14:24.058220 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:24.051721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.6.53.67:45291)
0620 07:14:24.051896 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:24.051903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:24.051948 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:14:24.052176 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:24.052180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:24.052265 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:24.052368 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:24.052763 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:24.053523 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:24.055415 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:24.055796 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:24.055887 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:24.056973 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:24.056980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:24.056994 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:24.057515 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:24.057566 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:24.057584 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:24.057672 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:24.057827 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 07:14:24.057907 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:14:24.059108 26671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:24.109202 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:24.108782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37849 (local address 127.0.0.1:53920)
0620 07:14:24.108966 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:24.109059 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:37849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 07:14:24.110391 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:24.109911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.65:37849 (local address 127.0.0.1:53924)
0620 07:14:24.110139 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:24.110232 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.65:37849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 07:14:24.147131 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 26549
I20260620 07:14:24.163293 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 26198
I20260620 07:14:24.164434 26259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:24.271860 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 26198
2026-06-20T07:14:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2748 ms)
[----------] 6 tests from EncryptionPolicyTest (21324 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:24 dist-test-slave-4n6h krb5kdc[26729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:24 dist-test-slave-4n6h krb5kdc[26729](info): set up 2 sockets
Jun 20 07:14:24 dist-test-slave-4n6h krb5kdc[26729](info): commencing operation
krb5kdc: starting...
W20260620 07:14:25.390753 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:25 dist-test-slave-4n6h krb5kdc[26729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:14:25.638927 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44745
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:25.963148 26742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:25.963413 26742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:25.963570 26742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:25.970739 26742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:25.970858 26742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:25.970949 26742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:25.971048 26742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:25.971123 26742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:25.980676 26742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44745
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:25.981835 26742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:25.983095 26742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:25.994836 26747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:25.994925 26748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:25.995538 26750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:25.995595 26742 server_base.cc:1034] running on GCE node
I20260620 07:14:25.996335 26742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:25.997644 26742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:25.998867 26742 hybrid_clock.cc:648] HybridClock initialized: now 1781939665998816 us; error 65 us; skew 500 ppm
I20260620 07:14:25.999228 26742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:25 dist-test-slave-4n6h krb5kdc[26729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:26.002184 26742 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:14:26.003574 26742 webserver.cc:460] Webserver started at http://127.0.0.1:37697/ using document root <none> and password file <none>
I20260620 07:14:26.004228 26742 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:26.004387 26742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:26.004707 26742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:26.007666 26742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "4732c707362a42d594f310cd4c862281"
format_stamp: "Formatted at 2026-06-20 07:14:26 on dist-test-slave-4n6h"
server_key: "ed51630edcf8aaf2bebd683ef17adeaf"
server_key_iv: "ed7364ffbf68003b45c6b59279a4c7a8"
server_key_version: "encryptionkey@0"
I20260620 07:14:26.008440 26742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "4732c707362a42d594f310cd4c862281"
format_stamp: "Formatted at 2026-06-20 07:14:26 on dist-test-slave-4n6h"
server_key: "ed51630edcf8aaf2bebd683ef17adeaf"
server_key_iv: "ed7364ffbf68003b45c6b59279a4c7a8"
server_key_version: "encryptionkey@0"
I20260620 07:14:26.013475 26742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 07:14:26.016952 26757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:26.018222 26742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:14:26.018457 26742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "4732c707362a42d594f310cd4c862281"
format_stamp: "Formatted at 2026-06-20 07:14:26 on dist-test-slave-4n6h"
server_key: "ed51630edcf8aaf2bebd683ef17adeaf"
server_key_iv: "ed7364ffbf68003b45c6b59279a4c7a8"
server_key_version: "encryptionkey@0"
I20260620 07:14:26.018656 26742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:26.031410 26742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:26.035097 26742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:26.035359 26742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:26.053786 26742 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44745
I20260620 07:14:26.053828 26808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44745 every 8 connection(s)
I20260620 07:14:26.055123 26742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:26.058712 26809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:26.061826 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 26742
I20260620 07:14:26.062058 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:26.062763 6357 external_mini_cluster.cc:1428] Setting key c77b4924f6d280d894974214db50f485
I20260620 07:14:26.071461 26809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281: Bootstrap starting.
I20260620 07:14:26.074923 26809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:26.075868 26809 log.cc:824] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:14:26 dist-test-slave-4n6h krb5kdc[26729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:14:26.078764 26809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281: No bootstrap required, opened a new log
I20260620 07:14:26.086650 26812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:26.064614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51749 (local address 127.0.0.1:44745)
0620 07:14:26.065397 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:26.065410 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:26.065457 (+ 47us) server_negotiation.cc:407] Connection header received
0620 07:14:26.067387 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:26.067414 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:26.067907 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:26.068315 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:26.069589 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:26.071494 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:26.072780 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:26.073793 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:26.073983 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:26.078130 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:26.078144 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:26.078162 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:26.078202 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:26.082173 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:26.082932 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:26.082942 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:26.082950 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:26.083165 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:26.083587 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:26.083592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:26.083595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:26.084262 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:26.084573 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:26.084957 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 07:14:26.085296 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":336,"threads_started":1}
I20260620 07:14:26.089395 26809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4732c707362a42d594f310cd4c862281" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44745 } }
I20260620 07:14:26.089799 26809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:26.089901 26809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4732c707362a42d594f310cd4c862281, State: Initialized, Role: FOLLOWER
I20260620 07:14:26.090569 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:26.090555 26809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4732c707362a42d594f310cd4c862281" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44745 } }
I20260620 07:14:26.090822 26809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:26.090955 26809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:26.091120 26809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:26.093155 26809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4732c707362a42d594f310cd4c862281" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44745 } }
I20260620 07:14:26.093636 26809 leader_election.cc:302] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4732c707362a42d594f310cd4c862281; no voters:
I20260620 07:14:26.094357 26809 leader_election.cc:288] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:26.094501 26818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:26.096544 26809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:26.096593 26818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [term 1 LEADER]: Becoming Leader. State: Replica: 4732c707362a42d594f310cd4c862281, State: Running, Role: LEADER
I20260620 07:14:26.097287 26818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4732c707362a42d594f310cd4c862281" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44745 } }
I20260620 07:14:26.100315 26819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4732c707362a42d594f310cd4c862281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4732c707362a42d594f310cd4c862281" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44745 } } }
I20260620 07:14:26.100761 26819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:26.101312 26820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4732c707362a42d594f310cd4c862281. Latest consensus state: current_term: 1 leader_uuid: "4732c707362a42d594f310cd4c862281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4732c707362a42d594f310cd4c862281" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44745 } } }
I20260620 07:14:26.101598 26820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:26.109010 26827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:26.114758 26827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:26.115223 26812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:26.095436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60631 (local address 127.0.0.1:44745)
0620 07:14:26.095589 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:26.095594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:26.096356 (+ 762us) server_negotiation.cc:407] Connection header received
0620 07:14:26.096665 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:26.096671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:26.096752 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:26.096815 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:26.099045 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:26.100556 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:26.102836 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:26.103735 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:26.103863 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:26.107297 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:26.107302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:26.107305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:26.107346 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:26.110962 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:26.112019 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:26.112025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:26.112028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:26.112175 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:26.112732 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:26.112736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:26.112739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:26.113172 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:26.113468 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:26.114688 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0620 07:14:26.114878 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:14:26.125517 26827 catalog_manager.cc:1347] Generated new cluster ID: 080a15620fb440b3b659af2b3b29c08d
I20260620 07:14:26.125703 26827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:26.138918 26827 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:26.140195 26827 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:26.150153 26827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4732c707362a42d594f310cd4c862281: Generated new TSK 0
I20260620 07:14:26.150950 26827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:26.156708 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 26742
I20260620 07:14:26.158565 26804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:26.251876 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 26742
W20260620 07:14:26.268666 26727 connection.cc:535] client connection to 127.0.0.1:44745 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44745): Connection reset by peer (error 104)
2026-06-20T07:14:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:26 dist-test-slave-4n6h krb5kdc[26841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:26 dist-test-slave-4n6h krb5kdc[26841](info): set up 2 sockets
Jun 20 07:14:26 dist-test-slave-4n6h krb5kdc[26841](info): commencing operation
krb5kdc: starting...
W20260620 07:14:27.354660 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:27 dist-test-slave-4n6h krb5kdc[26841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:14:27.601219 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:27.923121 26854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:27.923446 26854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:27.923611 26854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:27.930778 26854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:27.930929 26854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:27.931030 26854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:27.931141 26854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:27.931229 26854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:27.940971 26854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44269
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:27.942273 26854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:27.943558 26854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:27.955190 26859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:27.955413 26860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:27.955554 26862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:27.956678 26854 server_base.cc:1034] running on GCE node
I20260620 07:14:27.957406 26854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:27.958884 26854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:27.960078 26854 hybrid_clock.cc:648] HybridClock initialized: now 1781939667960041 us; error 29 us; skew 500 ppm
I20260620 07:14:27.960407 26854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:27 dist-test-slave-4n6h krb5kdc[26841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:27.963440 26854 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:14:27.964898 26854 webserver.cc:460] Webserver started at http://127.0.0.1:39375/ using document root <none> and password file <none>
I20260620 07:14:27.965574 26854 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:27.965696 26854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:27.966014 26854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:27.968863 26854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "d1a283a520c8444b8e7adda82d7a0767"
format_stamp: "Formatted at 2026-06-20 07:14:27 on dist-test-slave-4n6h"
server_key: "a7bf166208edab7bf209b43743d31098"
server_key_iv: "9e67a2271b6fc7cefe5fb57a907ede0e"
server_key_version: "encryptionkey@0"
I20260620 07:14:27.969581 26854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "d1a283a520c8444b8e7adda82d7a0767"
format_stamp: "Formatted at 2026-06-20 07:14:27 on dist-test-slave-4n6h"
server_key: "a7bf166208edab7bf209b43743d31098"
server_key_iv: "9e67a2271b6fc7cefe5fb57a907ede0e"
server_key_version: "encryptionkey@0"
I20260620 07:14:27.974478 26854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260620 07:14:27.977738 26869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:27.979033 26854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:27.979216 26854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "d1a283a520c8444b8e7adda82d7a0767"
format_stamp: "Formatted at 2026-06-20 07:14:27 on dist-test-slave-4n6h"
server_key: "a7bf166208edab7bf209b43743d31098"
server_key_iv: "9e67a2271b6fc7cefe5fb57a907ede0e"
server_key_version: "encryptionkey@0"
I20260620 07:14:27.979403 26854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:28.004819 26854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:28.009012 26854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:28.009311 26854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:28.028226 26854 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44269
I20260620 07:14:28.028311 26920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44269 every 8 connection(s)
I20260620 07:14:28.029542 26854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:28.033367 26921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:28.033641 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 26854
I20260620 07:14:28.033823 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:28.034240 6357 external_mini_cluster.cc:1428] Setting key 8d953c4822c78151d8239e1d69f93ab2
I20260620 07:14:28.047405 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:28.036006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58567 (local address 127.0.0.1:44269)
0620 07:14:28.036875 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:28.036886 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:28.036927 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:14:28.038499 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:28.038519 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:28.038902 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:28.039233 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:28.039884 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:28.041157 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:28.043115 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:28.043745 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:28.043882 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:28.045193 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:28.045202 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:28.045214 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:28.045735 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:28.045809 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:28.045832 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:28.045953 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:28.046317 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 07:14:28.046501 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":451,"threads_started":1}
I20260620 07:14:28.047498 26921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767: Bootstrap starting.
I20260620 07:14:28.050694 26921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:28.051054 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:28.051755 26921 log.cc:824] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:28.054800 26921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767: No bootstrap required, opened a new log
I20260620 07:14:28.062192 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:28.055553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38879 (local address 127.0.0.1:44269)
0620 07:14:28.055725 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:28.055731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:28.056208 (+ 477us) server_negotiation.cc:407] Connection header received
0620 07:14:28.056355 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:28.056361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:28.056448 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:28.056569 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:28.057002 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:28.058174 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:28.059492 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:28.060248 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:28.060398 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:28.061423 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:28.061432 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:28.061441 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:28.061462 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:28.061514 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:28.061533 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:28.061643 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:28.061808 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 07:14:28.061857 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:14:28.064317 26921 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a283a520c8444b8e7adda82d7a0767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44269 } }
I20260620 07:14:28.064672 26921 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:28.064795 26921 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a283a520c8444b8e7adda82d7a0767, State: Initialized, Role: FOLLOWER
I20260620 07:14:28.065393 26921 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a283a520c8444b8e7adda82d7a0767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44269 } }
I20260620 07:14:28.065626 26921 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:28.065769 26921 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:28.066111 26921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:28.068886 26921 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a283a520c8444b8e7adda82d7a0767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44269 } }
I20260620 07:14:28.069458 26921 leader_election.cc:302] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1a283a520c8444b8e7adda82d7a0767; no voters:
I20260620 07:14:28.070039 26921 leader_election.cc:288] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:28.070142 26931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:28.071003 26931 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [term 1 LEADER]: Becoming Leader. State: Replica: d1a283a520c8444b8e7adda82d7a0767, State: Running, Role: LEADER
I20260620 07:14:28.071514 26931 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a283a520c8444b8e7adda82d7a0767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44269 } }
I20260620 07:14:28.072446 26921 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:28.074573 26932 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1a283a520c8444b8e7adda82d7a0767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a283a520c8444b8e7adda82d7a0767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44269 } } }
I20260620 07:14:28.075240 26932 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:28.074437 26933 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1a283a520c8444b8e7adda82d7a0767. Latest consensus state: current_term: 1 leader_uuid: "d1a283a520c8444b8e7adda82d7a0767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a283a520c8444b8e7adda82d7a0767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44269 } } }
I20260620 07:14:28.075837 26933 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:28.076946 26939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:28.081480 26939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:28.089866 26939 catalog_manager.cc:1347] Generated new cluster ID: 10a2b173bc9a4df3a9bfb70c7bae13bf
I20260620 07:14:28.089989 26939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:28.101066 26939 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:28.102402 26939 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:28.117386 26939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1a283a520c8444b8e7adda82d7a0767: Generated new TSK 0
I20260620 07:14:28.118489 26939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:28.133260 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 26854
W20260620 07:14:28.134487 26916 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58567
W20260620 07:14:28.135255 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 26854: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:14:28.135468 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 26854
W20260620 07:14:28.152077 26839 connection.cc:535] client connection to 127.0.0.1:44269 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44269): Connection reset by peer (error 104)
2026-06-20T07:14:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1882 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:28 dist-test-slave-4n6h krb5kdc[26952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:28 dist-test-slave-4n6h krb5kdc[26952](info): set up 2 sockets
Jun 20 07:14:28 dist-test-slave-4n6h krb5kdc[26952](info): commencing operation
krb5kdc: starting...
W20260620 07:14:29.238492 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:29 dist-test-slave-4n6h krb5kdc[26952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:14:29.489236 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:29.812194 26965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:29.812484 26965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:29.812644 26965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:29.819731 26965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:29.819842 26965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:29.819969 26965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:29.820073 26965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:29.820137 26965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:29.829932 26965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:29.831118 26965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:29.832348 26965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:29.846387 26973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:29.846431 26971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:29.846457 26970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:29.847877 26965 server_base.cc:1034] running on GCE node
I20260620 07:14:29.848838 26965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:29.850587 26965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:29.851827 26965 hybrid_clock.cc:648] HybridClock initialized: now 1781939669851777 us; error 49 us; skew 500 ppm
I20260620 07:14:29.852289 26965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:29 dist-test-slave-4n6h krb5kdc[26952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:29.860541 26965 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:14:29.862170 26965 webserver.cc:460] Webserver started at http://127.0.0.1:37845/ using document root <none> and password file <none>
I20260620 07:14:29.862895 26965 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:29.863003 26965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:29.863250 26965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:29.866151 26965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "26d5d57019124739a3f032e8c95f225b"
format_stamp: "Formatted at 2026-06-20 07:14:29 on dist-test-slave-4n6h"
server_key: "3a41507e75a6d1a7f5b8ef650f05dda2"
server_key_iv: "b8adfd51570fc435512b22d7f68bee26"
server_key_version: "encryptionkey@0"
I20260620 07:14:29.866951 26965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "26d5d57019124739a3f032e8c95f225b"
format_stamp: "Formatted at 2026-06-20 07:14:29 on dist-test-slave-4n6h"
server_key: "3a41507e75a6d1a7f5b8ef650f05dda2"
server_key_iv: "b8adfd51570fc435512b22d7f68bee26"
server_key_version: "encryptionkey@0"
I20260620 07:14:29.871846 26965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:14:29.875145 26980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:29.876415 26965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:14:29.876559 26965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "26d5d57019124739a3f032e8c95f225b"
format_stamp: "Formatted at 2026-06-20 07:14:29 on dist-test-slave-4n6h"
server_key: "3a41507e75a6d1a7f5b8ef650f05dda2"
server_key_iv: "b8adfd51570fc435512b22d7f68bee26"
server_key_version: "encryptionkey@0"
I20260620 07:14:29.876729 26965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:29.882547 26965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:29.886669 26965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:29.886914 26965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:29.905560 26965 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44261
I20260620 07:14:29.905632 27031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44261 every 8 connection(s)
I20260620 07:14:29.907086 26965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:29.910687 27032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:29.911890 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 26965
I20260620 07:14:29.912087 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:29.912621 6357 external_mini_cluster.cc:1428] Setting key 106b7a545f8cfb8ddf92c54f252ff788
I20260620 07:14:29.920867 27035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:29.914477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53241 (local address 127.0.0.1:44261)
0620 07:14:29.915379 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:29.915393 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:29.915433 (+ 40us) server_negotiation.cc:407] Connection header received
0620 07:14:29.917004 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:29.917029 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:29.917345 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:29.917662 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:29.918787 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:29.918794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:29.918847 (+ 53us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:29.918862 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:29.918954 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:29.919000 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:29.919235 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:29.919386 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 07:14:29.919578 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":456,"threads_started":1}
I20260620 07:14:29.924978 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:29.925535 27032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b: Bootstrap starting.
I20260620 07:14:29.929380 27032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:29.930701 27032 log.cc:824] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:29.932917 27035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:29.930381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50091 (local address 127.0.0.1:44261)
0620 07:14:29.930573 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:29.930579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:29.930938 (+ 359us) server_negotiation.cc:407] Connection header received
0620 07:14:29.931089 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:29.931095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:29.931153 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:29.931281 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:29.932271 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:29.932277 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:29.932281 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:29.932298 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:29.932339 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:29.932358 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:29.932471 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:29.932597 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 07:14:29.932652 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:14:29.933795 27032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b: No bootstrap required, opened a new log
I20260620 07:14:29.940858 27032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d5d57019124739a3f032e8c95f225b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44261 } }
I20260620 07:14:29.941241 27032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:29.941360 27032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d5d57019124739a3f032e8c95f225b, State: Initialized, Role: FOLLOWER
I20260620 07:14:29.942028 27032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d5d57019124739a3f032e8c95f225b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44261 } }
I20260620 07:14:29.942278 27032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:29.942438 27032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:29.942600 27032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:29.944693 27032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d5d57019124739a3f032e8c95f225b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44261 } }
I20260620 07:14:29.945267 27032 leader_election.cc:302] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26d5d57019124739a3f032e8c95f225b; no voters:
I20260620 07:14:29.945833 27032 leader_election.cc:288] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:29.946004 27042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:29.946759 27042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [term 1 LEADER]: Becoming Leader. State: Replica: 26d5d57019124739a3f032e8c95f225b, State: Running, Role: LEADER
I20260620 07:14:29.947265 27042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d5d57019124739a3f032e8c95f225b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44261 } }
I20260620 07:14:29.947960 27032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:29.949769 27044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26d5d57019124739a3f032e8c95f225b. Latest consensus state: current_term: 1 leader_uuid: "26d5d57019124739a3f032e8c95f225b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d5d57019124739a3f032e8c95f225b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44261 } } }
I20260620 07:14:29.950114 27044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:29.950470 27043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26d5d57019124739a3f032e8c95f225b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d5d57019124739a3f032e8c95f225b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44261 } } }
I20260620 07:14:29.950944 27043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:29.952744 27048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:29.959045 27048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:29.968791 27048 catalog_manager.cc:1347] Generated new cluster ID: bb2310aec1ee47128e47d0941255d4f9
I20260620 07:14:29.968894 27048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:30.017824 27048 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:30.018363 27048 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:30.025014 27048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26d5d57019124739a3f032e8c95f225b: Generated new TSK 0
I20260620 07:14:30.025763 27048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:30.037662 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 26965
W20260620 07:14:30.038923 27027 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53241
W20260620 07:14:30.039726 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 26965: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:14:30.039891 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 26965
2026-06-20T07:14:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1901 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:30 dist-test-slave-4n6h krb5kdc[27063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:30 dist-test-slave-4n6h krb5kdc[27063](info): set up 2 sockets
Jun 20 07:14:30 dist-test-slave-4n6h krb5kdc[27063](info): commencing operation
krb5kdc: starting...
W20260620 07:14:31.142797 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:31 dist-test-slave-4n6h krb5kdc[27063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:14:31.382098 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33677
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:31.698778 27076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:31.699049 27076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:31.699218 27076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:31.706336 27076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:31.706445 27076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:31.706629 27076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:31.706748 27076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:31.706835 27076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:31.716351 27076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:31.717624 27076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:31.718930 27076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:31.730482 27081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:31.730769 27084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:31.731105 27082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:31.731580 27076 server_base.cc:1034] running on GCE node
I20260620 07:14:31.732336 27076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:31.733594 27076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:31.734810 27076 hybrid_clock.cc:648] HybridClock initialized: now 1781939671734771 us; error 54 us; skew 500 ppm
I20260620 07:14:31.735143 27076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:31 dist-test-slave-4n6h krb5kdc[27063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:31.738226 27076 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:14:31.739591 27076 webserver.cc:460] Webserver started at http://127.0.0.1:36909/ using document root <none> and password file <none>
I20260620 07:14:31.740193 27076 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:31.740286 27076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:31.740520 27076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:31.743350 27076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "1a4800eabd124b12870f5aff5f274f9a"
format_stamp: "Formatted at 2026-06-20 07:14:31 on dist-test-slave-4n6h"
server_key: "917d3578f09d35e85d24da1b41d6170e"
server_key_iv: "1651100e6b2684af831cf1143df9dffc"
server_key_version: "encryptionkey@0"
I20260620 07:14:31.744097 27076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "1a4800eabd124b12870f5aff5f274f9a"
format_stamp: "Formatted at 2026-06-20 07:14:31 on dist-test-slave-4n6h"
server_key: "917d3578f09d35e85d24da1b41d6170e"
server_key_iv: "1651100e6b2684af831cf1143df9dffc"
server_key_version: "encryptionkey@0"
I20260620 07:14:31.749001 27076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:14:31.752348 27091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:31.753599 27076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:14:31.753731 27076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "1a4800eabd124b12870f5aff5f274f9a"
format_stamp: "Formatted at 2026-06-20 07:14:31 on dist-test-slave-4n6h"
server_key: "917d3578f09d35e85d24da1b41d6170e"
server_key_iv: "1651100e6b2684af831cf1143df9dffc"
server_key_version: "encryptionkey@0"
I20260620 07:14:31.753894 27076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:31.772516 27076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:31.776445 27076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:31.776677 27076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:31.794716 27076 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33677
I20260620 07:14:31.794824 27142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33677 every 8 connection(s)
I20260620 07:14:31.796036 27076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:31.799536 27143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:31.805303 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 27076
I20260620 07:14:31.805488 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:31.805944 6357 external_mini_cluster.cc:1428] Setting key bb571f52dab71fc2770ef0316bfc3d24
I20260620 07:14:31.809717 27143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a: Bootstrap starting.
I20260620 07:14:31.813311 27143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:31.814431 27143 log.cc:824] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:31.817186 27143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a: No bootstrap required, opened a new log
Jun 20 07:14:31 dist-test-slave-4n6h krb5kdc[27063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:14:31.825212 27143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4800eabd124b12870f5aff5f274f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } }
I20260620 07:14:31.825584 27143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:31.825706 27143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a4800eabd124b12870f5aff5f274f9a, State: Initialized, Role: FOLLOWER
I20260620 07:14:31.826309 27143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4800eabd124b12870f5aff5f274f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } }
I20260620 07:14:31.826584 27143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:31.826748 27143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:31.826932 27143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:31.828281 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:31.807716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41051 (local address 127.0.0.1:33677)
0620 07:14:31.808530 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:31.808540 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:31.808576 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:14:31.810189 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:31.810209 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:31.810604 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:31.810886 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:31.812668 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:31.813995 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:31.815622 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:31.816242 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:31.816380 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:31.821049 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:31.821072 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:31.821084 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:31.821121 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:31.824222 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:31.824913 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:31.824925 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:31.824930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:31.825078 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:31.825453 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:31.825458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:31.825468 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:31.826165 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:31.826521 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:31.826947 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 07:14:31.827329 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":359,"threads_started":1}
I20260620 07:14:31.829325 27143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4800eabd124b12870f5aff5f274f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } }
I20260620 07:14:31.829859 27143 leader_election.cc:302] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a4800eabd124b12870f5aff5f274f9a; no voters:
I20260620 07:14:31.830642 27143 leader_election.cc:288] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:31.830749 27148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:31.832149 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:31.832213 27148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [term 1 LEADER]: Becoming Leader. State: Replica: 1a4800eabd124b12870f5aff5f274f9a, State: Running, Role: LEADER
I20260620 07:14:31.832751 27143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:31.832723 27148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4800eabd124b12870f5aff5f274f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } }
I20260620 07:14:31.839600 27149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a4800eabd124b12870f5aff5f274f9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4800eabd124b12870f5aff5f274f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } } }
I20260620 07:14:31.840009 27149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:31.840523 27150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a4800eabd124b12870f5aff5f274f9a. Latest consensus state: current_term: 1 leader_uuid: "1a4800eabd124b12870f5aff5f274f9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4800eabd124b12870f5aff5f274f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } } }
I20260620 07:14:31.840791 27150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:31.848587 27156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:31.854971 27156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:31.870605 27146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:31.849818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34621 (local address 127.0.0.1:33677)
0620 07:14:31.849996 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:31.850002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:31.850460 (+ 458us) server_negotiation.cc:407] Connection header received
0620 07:14:31.850726 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:31.850733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:31.850843 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:31.850960 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:31.853047 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:31.854193 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:31.856160 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:31.856866 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:31.856961 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:31.863058 (+ 6097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:31.863066 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:31.863071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:31.863115 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:31.867022 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:31.868003 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:31.868008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:31.868012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:31.868199 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:31.868677 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:31.868682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:31.868685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:31.869108 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:31.869340 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:31.869953 (+ 613us) server_negotiation.cc:299] Negotiation successful
0620 07:14:31.870214 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:14:31.870218 27170 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:14:31.872660 27156 catalog_manager.cc:1347] Generated new cluster ID: 7d6b78e5331c4137b1b67ff04c74ff04
I20260620 07:14:31.872776 27156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:31.897619 27156 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:31.898969 27156 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:31.914165 27156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a4800eabd124b12870f5aff5f274f9a: Generated new TSK 0
I20260620 07:14:31.915066 27156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:31.929662 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 27076
I20260620 07:14:31.930833 27138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:32.023300 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 27076
2026-06-20T07:14:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1986 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:32 dist-test-slave-4n6h krb5kdc[27175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:32 dist-test-slave-4n6h krb5kdc[27175](info): set up 2 sockets
Jun 20 07:14:32 dist-test-slave-4n6h krb5kdc[27175](info): commencing operation
krb5kdc: starting...
W20260620 07:14:33.122618 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:33 dist-test-slave-4n6h krb5kdc[27175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:14:33.362329 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:33.686354 27188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:33.686635 27188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:33.686734 27188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:33.693921 27188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:33.694029 27188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:33.694075 27188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:33.694136 27188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:33.694182 27188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:33.703797 27188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:33.704973 27188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:33.706195 27188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:33.718019 27193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:33.718266 27194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:33.718237 27196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:33.719159 27188 server_base.cc:1034] running on GCE node
I20260620 07:14:33.720054 27188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:33.721383 27188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:33.722594 27188 hybrid_clock.cc:648] HybridClock initialized: now 1781939673722551 us; error 41 us; skew 500 ppm
I20260620 07:14:33.722970 27188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:33 dist-test-slave-4n6h krb5kdc[27175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:33.726353 27188 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:14:33.727877 27188 webserver.cc:460] Webserver started at http://127.0.0.1:37547/ using document root <none> and password file <none>
I20260620 07:14:33.728585 27188 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:33.728744 27188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:33.729061 27188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:33.732191 27188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "a1de475fdacb497b822e2b617b9010a7"
format_stamp: "Formatted at 2026-06-20 07:14:33 on dist-test-slave-4n6h"
server_key: "249a59af11ba2dafb6a9a4c184be0df3"
server_key_iv: "53845f37396778bd58e6cfabaae1bf2b"
server_key_version: "encryptionkey@0"
I20260620 07:14:33.732992 27188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "a1de475fdacb497b822e2b617b9010a7"
format_stamp: "Formatted at 2026-06-20 07:14:33 on dist-test-slave-4n6h"
server_key: "249a59af11ba2dafb6a9a4c184be0df3"
server_key_iv: "53845f37396778bd58e6cfabaae1bf2b"
server_key_version: "encryptionkey@0"
I20260620 07:14:33.738405 27188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 07:14:33.741891 27203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:33.743188 27188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:33.743366 27188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "a1de475fdacb497b822e2b617b9010a7"
format_stamp: "Formatted at 2026-06-20 07:14:33 on dist-test-slave-4n6h"
server_key: "249a59af11ba2dafb6a9a4c184be0df3"
server_key_iv: "53845f37396778bd58e6cfabaae1bf2b"
server_key_version: "encryptionkey@0"
I20260620 07:14:33.743556 27188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:33.767267 27188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:33.771871 27188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:33.772176 27188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:33.791895 27188 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36807
I20260620 07:14:33.791926 27254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36807 every 8 connection(s)
I20260620 07:14:33.793308 27188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:33.795228 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 27188
I20260620 07:14:33.795418 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:33.795879 6357 external_mini_cluster.cc:1428] Setting key 0eb073853b9007859c838eebae9427d9
I20260620 07:14:33.797485 27255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:33.808688 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:33.797619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33815 (local address 127.0.0.1:36807)
0620 07:14:33.798399 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:33.798410 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:33.798448 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:14:33.800041 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:33.800060 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:33.800427 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:33.800734 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:33.801366 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:33.802986 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:33.804431 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:33.805103 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:33.805301 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:33.806516 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:33.806527 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:33.806540 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:33.807087 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:33.807155 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:33.807188 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:33.807334 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:33.807526 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 07:14:33.807780 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":330,"threads_started":1}
I20260620 07:14:33.809490 27255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7: Bootstrap starting.
I20260620 07:14:33.812132 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:33.812747 27255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:33.813681 27255 log.cc:824] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:33.816290 27255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7: No bootstrap required, opened a new log
I20260620 07:14:33.823390 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:33.816549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47393 (local address 127.0.0.1:36807)
0620 07:14:33.816792 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:33.816799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:33.817177 (+ 378us) server_negotiation.cc:407] Connection header received
0620 07:14:33.817441 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:33.817452 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:33.817515 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:33.817612 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:33.818324 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:33.819069 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:33.820914 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:33.821347 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:33.821443 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:33.822619 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:33.822626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:33.822629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:33.822644 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:33.822678 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:33.822696 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:33.822766 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:33.822932 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 07:14:33.822999 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:14:33.825812 27255 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1de475fdacb497b822e2b617b9010a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36807 } }
I20260620 07:14:33.826185 27255 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:33.826355 27255 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1de475fdacb497b822e2b617b9010a7, State: Initialized, Role: FOLLOWER
I20260620 07:14:33.826901 27255 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1de475fdacb497b822e2b617b9010a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36807 } }
I20260620 07:14:33.827134 27255 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:33.827282 27255 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:33.827451 27255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:33.829632 27255 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1de475fdacb497b822e2b617b9010a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36807 } }
I20260620 07:14:33.830103 27255 leader_election.cc:302] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1de475fdacb497b822e2b617b9010a7; no voters:
I20260620 07:14:33.830648 27255 leader_election.cc:288] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:33.830749 27265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:33.831480 27265 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [term 1 LEADER]: Becoming Leader. State: Replica: a1de475fdacb497b822e2b617b9010a7, State: Running, Role: LEADER
I20260620 07:14:33.831983 27265 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1de475fdacb497b822e2b617b9010a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36807 } }
I20260620 07:14:33.832681 27255 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:33.840787 27267 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1de475fdacb497b822e2b617b9010a7. Latest consensus state: current_term: 1 leader_uuid: "a1de475fdacb497b822e2b617b9010a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1de475fdacb497b822e2b617b9010a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36807 } } }
I20260620 07:14:33.841253 27267 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:33.846622 27266 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1de475fdacb497b822e2b617b9010a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1de475fdacb497b822e2b617b9010a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36807 } } }
I20260620 07:14:33.847260 27266 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:33.847523 27272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:33.853714 27272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:33.862277 27272 catalog_manager.cc:1347] Generated new cluster ID: 4b7758a9983a4b7ab6c632536c8f0960
I20260620 07:14:33.862393 27272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:33.885308 27272 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:33.886739 27272 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:33.896219 27272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1de475fdacb497b822e2b617b9010a7: Generated new TSK 0
I20260620 07:14:33.897010 27272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:33.905130 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 27188
W20260620 07:14:33.906201 27250 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33815
W20260620 07:14:33.906908 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 27188: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:14:33.907058 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 27188
2026-06-20T07:14:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1882 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:34 dist-test-slave-4n6h krb5kdc[27286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:34 dist-test-slave-4n6h krb5kdc[27286](info): set up 2 sockets
Jun 20 07:14:34 dist-test-slave-4n6h krb5kdc[27286](info): commencing operation
krb5kdc: starting...
W20260620 07:14:35.002519 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:35 dist-test-slave-4n6h krb5kdc[27286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:14:35.261457 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42019
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:35.578610 27299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:35.579180 27299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:35.579386 27299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:35.586624 27299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:35.586766 27299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:35.586876 27299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:35.586985 27299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:35.587064 27299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:35.596689 27299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:35.597898 27299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:35.599236 27299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:35.610991 27304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:35.611104 27305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:35.611609 27307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:35.611656 27299 server_base.cc:1034] running on GCE node
I20260620 07:14:35.612414 27299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:35.613739 27299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:35.614959 27299 hybrid_clock.cc:648] HybridClock initialized: now 1781939675614917 us; error 39 us; skew 500 ppm
I20260620 07:14:35.615331 27299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:35 dist-test-slave-4n6h krb5kdc[27286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:35.618372 27299 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:14:35.619764 27299 webserver.cc:460] Webserver started at http://127.0.0.1:44543/ using document root <none> and password file <none>
I20260620 07:14:35.620371 27299 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:35.620467 27299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:35.620694 27299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:35.623610 27299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "cbfe9ee8455745b3ae99b282fa82471b"
format_stamp: "Formatted at 2026-06-20 07:14:35 on dist-test-slave-4n6h"
server_key: "905a57f56770f242f465f908ebfab92f"
server_key_iv: "d1037dc7808f79a9200aedbbfbeac36b"
server_key_version: "encryptionkey@0"
I20260620 07:14:35.624305 27299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "cbfe9ee8455745b3ae99b282fa82471b"
format_stamp: "Formatted at 2026-06-20 07:14:35 on dist-test-slave-4n6h"
server_key: "905a57f56770f242f465f908ebfab92f"
server_key_iv: "d1037dc7808f79a9200aedbbfbeac36b"
server_key_version: "encryptionkey@0"
I20260620 07:14:35.629272 27299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:14:35.632680 27314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:35.633965 27299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 07:14:35.634135 27299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "cbfe9ee8455745b3ae99b282fa82471b"
format_stamp: "Formatted at 2026-06-20 07:14:35 on dist-test-slave-4n6h"
server_key: "905a57f56770f242f465f908ebfab92f"
server_key_iv: "d1037dc7808f79a9200aedbbfbeac36b"
server_key_version: "encryptionkey@0"
I20260620 07:14:35.634397 27299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:35.640053 27299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:35.644124 27299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:35.644408 27299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:35.662482 27299 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42019
I20260620 07:14:35.662555 27365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42019 every 8 connection(s)
I20260620 07:14:35.663794 27299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:35.667346 27366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:35.673638 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 27299
I20260620 07:14:35.673818 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:35.674389 6357 external_mini_cluster.cc:1428] Setting key ba707ddf4d5ad868de4fd322c1d09305
I20260620 07:14:35.678149 27366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b: Bootstrap starting.
I20260620 07:14:35.681372 27366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:35.682008 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:35.676133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57469 (local address 127.0.0.1:42019)
0620 07:14:35.676926 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:35.676937 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:35.676976 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:14:35.678741 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:35.678762 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:35.679065 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:35.679424 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:35.680413 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:35.680418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:35.680430 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:35.680443 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:35.680495 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:35.680516 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:35.680710 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:35.680856 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 07:14:35.681021 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":340,"threads_started":1}
I20260620 07:14:35.682515 27366 log.cc:824] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:35.685276 27366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b: No bootstrap required, opened a new log
I20260620 07:14:35.685552 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:35.692154 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:35.689783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49335 (local address 127.0.0.1:42019)
0620 07:14:35.689979 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:35.689989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:35.690337 (+ 348us) server_negotiation.cc:407] Connection header received
0620 07:14:35.690459 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:35.690463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:35.690508 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:35.690589 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:35.691465 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:35.691468 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:35.691471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:35.691488 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:35.691523 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:35.691540 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:35.691634 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:35.691803 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 07:14:35.691853 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:14:35.693730 27366 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfe9ee8455745b3ae99b282fa82471b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42019 } }
I20260620 07:14:35.694103 27366 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:35.694206 27366 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfe9ee8455745b3ae99b282fa82471b, State: Initialized, Role: FOLLOWER
I20260620 07:14:35.694761 27366 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfe9ee8455745b3ae99b282fa82471b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42019 } }
I20260620 07:14:35.695006 27366 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:35.695140 27366 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:35.695317 27366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:35.697455 27366 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfe9ee8455745b3ae99b282fa82471b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42019 } }
I20260620 07:14:35.697990 27366 leader_election.cc:302] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbfe9ee8455745b3ae99b282fa82471b; no voters:
I20260620 07:14:35.698683 27366 leader_election.cc:288] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:35.698848 27376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:35.699545 27376 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [term 1 LEADER]: Becoming Leader. State: Replica: cbfe9ee8455745b3ae99b282fa82471b, State: Running, Role: LEADER
I20260620 07:14:35.700166 27376 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfe9ee8455745b3ae99b282fa82471b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42019 } }
I20260620 07:14:35.700721 27366 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:35.702908 27378 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbfe9ee8455745b3ae99b282fa82471b. Latest consensus state: current_term: 1 leader_uuid: "cbfe9ee8455745b3ae99b282fa82471b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfe9ee8455745b3ae99b282fa82471b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42019 } } }
I20260620 07:14:35.703408 27378 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:35.703071 27377 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbfe9ee8455745b3ae99b282fa82471b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfe9ee8455745b3ae99b282fa82471b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42019 } } }
I20260620 07:14:35.703758 27377 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:35.706048 27384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:35.710850 27384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:35.719820 27384 catalog_manager.cc:1347] Generated new cluster ID: ee02594e8f334530892f31b7a1ebabdd
I20260620 07:14:35.719925 27384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:35.747278 27384 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:35.747749 27384 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:35.754371 27384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbfe9ee8455745b3ae99b282fa82471b: Generated new TSK 0
I20260620 07:14:35.755281 27384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:35.771150 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 27299
W20260620 07:14:35.772890 27361 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57469
W20260620 07:14:35.773617 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 27299: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:14:35.773777 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 27299
2026-06-20T07:14:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1867 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:35 dist-test-slave-4n6h krb5kdc[27397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:35 dist-test-slave-4n6h krb5kdc[27397](info): set up 2 sockets
Jun 20 07:14:35 dist-test-slave-4n6h krb5kdc[27397](info): commencing operation
krb5kdc: starting...
W20260620 07:14:36.890586 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:37 dist-test-slave-4n6h krb5kdc[27397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:14:37.135279 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35215
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:37.453541 27410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:37.453830 27410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:37.453920 27410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:37.461016 27410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:37.461125 27410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:37.461170 27410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:37.461236 27410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:37.461280 27410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:37.470831 27410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:37.471993 27410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:37.473166 27410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:37.484640 27415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:37.484709 27418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:37.484685 27416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:37.485515 27410 server_base.cc:1034] running on GCE node
I20260620 07:14:37.486374 27410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:37.487744 27410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:37.488977 27410 hybrid_clock.cc:648] HybridClock initialized: now 1781939677488934 us; error 33 us; skew 500 ppm
I20260620 07:14:37.489331 27410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:37 dist-test-slave-4n6h krb5kdc[27397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:37.492446 27410 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:14:37.493902 27410 webserver.cc:460] Webserver started at http://127.0.0.1:43639/ using document root <none> and password file <none>
I20260620 07:14:37.494650 27410 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:37.494799 27410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:37.495087 27410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:37.497881 27410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "e35478c7a64b4af086a0ef0987067534"
format_stamp: "Formatted at 2026-06-20 07:14:37 on dist-test-slave-4n6h"
server_key: "011186a6553da58af41b9737e1e4dd62"
server_key_iv: "e20900190d4970c3dcd1ab3434429988"
server_key_version: "encryptionkey@0"
I20260620 07:14:37.498669 27410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "e35478c7a64b4af086a0ef0987067534"
format_stamp: "Formatted at 2026-06-20 07:14:37 on dist-test-slave-4n6h"
server_key: "011186a6553da58af41b9737e1e4dd62"
server_key_iv: "e20900190d4970c3dcd1ab3434429988"
server_key_version: "encryptionkey@0"
I20260620 07:14:37.503473 27410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 07:14:37.506723 27425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:37.507907 27410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:14:37.508090 27410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "e35478c7a64b4af086a0ef0987067534"
format_stamp: "Formatted at 2026-06-20 07:14:37 on dist-test-slave-4n6h"
server_key: "011186a6553da58af41b9737e1e4dd62"
server_key_iv: "e20900190d4970c3dcd1ab3434429988"
server_key_version: "encryptionkey@0"
I20260620 07:14:37.508276 27410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:37.527344 27410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:37.531615 27410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:37.531899 27410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:37.550171 27410 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35215
I20260620 07:14:37.550205 27476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35215 every 8 connection(s)
I20260620 07:14:37.551565 27410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:37.554694 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 27410
I20260620 07:14:37.554893 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:37.555263 27477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:37.555359 6357 external_mini_cluster.cc:1428] Setting key 2b3bac8c7f178fa0de31bd1dcbcef748
I20260620 07:14:37.569347 27477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534: Bootstrap starting.
Jun 20 07:14:37 dist-test-slave-4n6h krb5kdc[27397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:14:37.572799 27477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:37.573890 27477 log.cc:824] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:37.577018 27477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534: No bootstrap required, opened a new log
I20260620 07:14:37.578658 27480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:37.557083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:39643 (local address 127.0.0.1:35215)
0620 07:14:37.557863 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:37.557878 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:37.557925 (+ 47us) server_negotiation.cc:407] Connection header received
0620 07:14:37.560147 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:37.560174 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:37.560661 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:37.561091 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:37.562342 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:37.564311 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:37.565619 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:37.566736 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:37.566927 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:37.571215 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:37.571222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:37.571235 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:37.571305 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:37.574571 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:37.575452 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:37.575458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:37.575464 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:37.575618 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:37.576105 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:37.576112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:37.576114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:37.576577 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:37.576798 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:37.577348 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 07:14:37.577667 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":305,"threads_started":1}
I20260620 07:14:37.582360 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:37.586654 27477 raft_consensus.cc:348] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35478c7a64b4af086a0ef0987067534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35215 } }
I20260620 07:14:37.587075 27477 raft_consensus.cc:374] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:37.587188 27477 raft_consensus.cc:729] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35478c7a64b4af086a0ef0987067534, State: Initialized, Role: FOLLOWER
I20260620 07:14:37.587886 27477 consensus_queue.cc:260] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35478c7a64b4af086a0ef0987067534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35215 } }
I20260620 07:14:37.588164 27477 raft_consensus.cc:388] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:37.588327 27477 raft_consensus.cc:482] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:37.588522 27477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:37.590794 27477 raft_consensus.cc:504] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35478c7a64b4af086a0ef0987067534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35215 } }
I20260620 07:14:37.591264 27477 leader_election.cc:302] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e35478c7a64b4af086a0ef0987067534; no voters:
I20260620 07:14:37.591899 27477 leader_election.cc:288] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:37.591998 27487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:37.594419 27477 sys_catalog.cc:564] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:37.598734 27487 raft_consensus.cc:686] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [term 1 LEADER]: Becoming Leader. State: Replica: e35478c7a64b4af086a0ef0987067534, State: Running, Role: LEADER
I20260620 07:14:37.599639 27487 consensus_queue.cc:237] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35478c7a64b4af086a0ef0987067534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35215 } }
I20260620 07:14:37.603659 27489 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e35478c7a64b4af086a0ef0987067534. Latest consensus state: current_term: 1 leader_uuid: "e35478c7a64b4af086a0ef0987067534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35478c7a64b4af086a0ef0987067534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35215 } } }
I20260620 07:14:37.604110 27489 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:37.604725 27488 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e35478c7a64b4af086a0ef0987067534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35478c7a64b4af086a0ef0987067534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35215 } } }
I20260620 07:14:37.605057 27488 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:37.607748 27480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:37.587768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:49259 (local address 127.0.0.1:35215)
0620 07:14:37.587948 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:37.587954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:37.588279 (+ 325us) server_negotiation.cc:407] Connection header received
0620 07:14:37.588468 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:37.588477 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:37.588551 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:37.588650 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:37.589961 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:37.590789 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:37.592146 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:37.592856 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:37.592968 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:37.595658 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:37.595662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:37.595665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:37.595694 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:37.602219 (+ 6525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:37.603732 (+ 1513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:37.603736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:37.603743 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:37.603901 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:37.604502 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:37.604507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:37.604510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:37.605014 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:37.606781 (+ 1767us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:37.607000 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 07:14:37.607234 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:14:37.614300 27501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:37.619760 27501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:37.627815 27501 catalog_manager.cc:1347] Generated new cluster ID: 34a7dc9d3cda429399e4c662b157c0cd
I20260620 07:14:37.627924 27501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:37.653093 27501 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:37.655064 27501 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:37.667781 27501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e35478c7a64b4af086a0ef0987067534: Generated new TSK 0
I20260620 07:14:37.668848 27501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:37.684280 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 27410
I20260620 07:14:37.685915 27472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:37.778059 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 27410
W20260620 07:14:37.795149 27395 connection.cc:535] client connection to 127.0.0.1:35215 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35215): Connection reset by peer (error 104)
2026-06-20T07:14:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2004 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:37 dist-test-slave-4n6h krb5kdc[27509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:37 dist-test-slave-4n6h krb5kdc[27509](info): set up 2 sockets
Jun 20 07:14:37 dist-test-slave-4n6h krb5kdc[27509](info): commencing operation
krb5kdc: starting...
W20260620 07:14:38.874680 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:39 dist-test-slave-4n6h krb5kdc[27509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:14:39.116766 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:39.434382 27522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:39.434693 27522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:39.434891 27522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:39.442373 27522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:39.442510 27522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:39.442619 27522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:39.442724 27522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:39.442806 27522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:39.452592 27522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:39.453858 27522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:39.455183 27522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:39.466692 27527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:39.466784 27530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:39.466776 27528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:39.467748 27522 server_base.cc:1034] running on GCE node
I20260620 07:14:39.468408 27522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:39.469681 27522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:39.470873 27522 hybrid_clock.cc:648] HybridClock initialized: now 1781939679470833 us; error 33 us; skew 500 ppm
I20260620 07:14:39.471208 27522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:39 dist-test-slave-4n6h krb5kdc[27509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:39.474120 27522 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:14:39.475476 27522 webserver.cc:460] Webserver started at http://127.0.0.1:34931/ using document root <none> and password file <none>
I20260620 07:14:39.476128 27522 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:39.476253 27522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:39.476555 27522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:39.479386 27522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "304bb9c83ba64c9a8bea481cd66865cc"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-4n6h"
server_key: "259ad6ba669a1d526746f19b7b3b1d24"
server_key_iv: "053169e0def93fb574d1ddf2809c7987"
server_key_version: "encryptionkey@0"
I20260620 07:14:39.480103 27522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "304bb9c83ba64c9a8bea481cd66865cc"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-4n6h"
server_key: "259ad6ba669a1d526746f19b7b3b1d24"
server_key_iv: "053169e0def93fb574d1ddf2809c7987"
server_key_version: "encryptionkey@0"
I20260620 07:14:39.484910 27522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:14:39.488144 27537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:39.489374 27522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:14:39.489542 27522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "304bb9c83ba64c9a8bea481cd66865cc"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-4n6h"
server_key: "259ad6ba669a1d526746f19b7b3b1d24"
server_key_iv: "053169e0def93fb574d1ddf2809c7987"
server_key_version: "encryptionkey@0"
I20260620 07:14:39.489724 27522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:39.515725 27522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:39.519600 27522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:39.519879 27522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:39.538028 27522 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36435
I20260620 07:14:39.538064 27588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36435 every 8 connection(s)
I20260620 07:14:39.539317 27522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:39.542812 27589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:39.549469 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 27522
I20260620 07:14:39.549640 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:39.550112 6357 external_mini_cluster.cc:1428] Setting key 0fb0fc904cb037784d6cdbb15111370e
I20260620 07:14:39.554081 27589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc: Bootstrap starting.
I20260620 07:14:39.557818 27589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:39.559092 27589 log.cc:824] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:39.562251 27589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc: No bootstrap required, opened a new log
I20260620 07:14:39.563542 27592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:39.551859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:41573 (local address 127.0.0.1:36435)
0620 07:14:39.552623 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:39.552634 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:39.552670 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:14:39.554348 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:39.554370 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:39.554850 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:39.555294 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:39.555852 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.557113 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.559109 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.560110 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.560321 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:39.561378 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:39.561385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:39.561398 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:39.561890 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:39.561961 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:39.561984 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:39.562098 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:39.562496 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 07:14:39.562673 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":335,"threads_started":1}
I20260620 07:14:39.567263 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:39.571493 27589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304bb9c83ba64c9a8bea481cd66865cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } }
I20260620 07:14:39.571934 27589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:39.572045 27589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304bb9c83ba64c9a8bea481cd66865cc, State: Initialized, Role: FOLLOWER
I20260620 07:14:39.572729 27589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304bb9c83ba64c9a8bea481cd66865cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } }
I20260620 07:14:39.573009 27589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:39.573144 27589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:39.573321 27589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:39.576160 27589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304bb9c83ba64c9a8bea481cd66865cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } }
I20260620 07:14:39.576753 27589 leader_election.cc:302] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 304bb9c83ba64c9a8bea481cd66865cc; no voters:
I20260620 07:14:39.577499 27589 leader_election.cc:288] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:39.577616 27599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:39.578776 27599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [term 1 LEADER]: Becoming Leader. State: Replica: 304bb9c83ba64c9a8bea481cd66865cc, State: Running, Role: LEADER
I20260620 07:14:39.579521 27592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:39.572506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:58487 (local address 127.0.0.1:36435)
0620 07:14:39.572767 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:39.572772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:39.573017 (+ 245us) server_negotiation.cc:407] Connection header received
0620 07:14:39.573215 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:39.573222 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:39.573310 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:39.573415 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:39.573908 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.575063 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.576291 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.576998 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.577149 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:39.578447 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:39.578456 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:39.578463 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:39.578492 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:39.578550 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:39.578578 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:39.578698 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:39.579085 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 07:14:39.579183 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260620 07:14:39.579542 27599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304bb9c83ba64c9a8bea481cd66865cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } }
I20260620 07:14:39.579842 27589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:39.582477 27600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "304bb9c83ba64c9a8bea481cd66865cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304bb9c83ba64c9a8bea481cd66865cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } } }
I20260620 07:14:39.582785 27600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:39.583223 27601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 304bb9c83ba64c9a8bea481cd66865cc. Latest consensus state: current_term: 1 leader_uuid: "304bb9c83ba64c9a8bea481cd66865cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304bb9c83ba64c9a8bea481cd66865cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } } }
I20260620 07:14:39.583523 27601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:39.583616 27606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:39.589774 27606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:39.598515 27606 catalog_manager.cc:1347] Generated new cluster ID: 608b8101935f4dc9a3362dc5f229c79c
I20260620 07:14:39.598621 27606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:39.615373 27606 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:39.616627 27606 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:39.630821 27606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 304bb9c83ba64c9a8bea481cd66865cc: Generated new TSK 0
I20260620 07:14:39.631603 27606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:39.637895 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 27522
W20260620 07:14:39.638952 27584 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.218:41573
W20260620 07:14:39.639662 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 27522: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:14:39.639819 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 27522
W20260620 07:14:39.656216 27507 connection.cc:535] client connection to 127.0.0.1:36435 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36435): Connection reset by peer (error 104)
2026-06-20T07:14:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1860 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:39 dist-test-slave-4n6h krb5kdc[27620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:39 dist-test-slave-4n6h krb5kdc[27620](info): set up 2 sockets
Jun 20 07:14:39 dist-test-slave-4n6h krb5kdc[27620](info): commencing operation
krb5kdc: starting...
W20260620 07:14:40.738566 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:40 dist-test-slave-4n6h krb5kdc[27620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:14:40.975844 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34993
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:41.291066 27633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:41.291342 27633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:41.291499 27633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:41.298552 27633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:41.298677 27633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:41.298771 27633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:41.298877 27633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:41.298987 27633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:41.308332 27633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:41.309470 27633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:41.310693 27633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:41.321978 27638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:41.322044 27639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:41.321941 27641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:41.322803 27633 server_base.cc:1034] running on GCE node
I20260620 07:14:41.323505 27633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:41.324793 27633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:41.325985 27633 hybrid_clock.cc:648] HybridClock initialized: now 1781939681325937 us; error 46 us; skew 500 ppm
I20260620 07:14:41.326445 27633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:41 dist-test-slave-4n6h krb5kdc[27620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:41.334343 27633 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:14:41.335824 27633 webserver.cc:460] Webserver started at http://127.0.0.1:37145/ using document root <none> and password file <none>
I20260620 07:14:41.336509 27633 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:41.336661 27633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:41.336966 27633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:41.339861 27633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "331f718d980343b58eebc288bf39feb4"
format_stamp: "Formatted at 2026-06-20 07:14:41 on dist-test-slave-4n6h"
server_key: "882ee75bb4985f050e94781a6d6cee4a"
server_key_iv: "ddadd7448c82f6de2c89f96a8eedae54"
server_key_version: "encryptionkey@0"
I20260620 07:14:41.340627 27633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "331f718d980343b58eebc288bf39feb4"
format_stamp: "Formatted at 2026-06-20 07:14:41 on dist-test-slave-4n6h"
server_key: "882ee75bb4985f050e94781a6d6cee4a"
server_key_iv: "ddadd7448c82f6de2c89f96a8eedae54"
server_key_version: "encryptionkey@0"
I20260620 07:14:41.345592 27633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:14:41.348948 27648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:41.350121 27633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:14:41.350340 27633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "331f718d980343b58eebc288bf39feb4"
format_stamp: "Formatted at 2026-06-20 07:14:41 on dist-test-slave-4n6h"
server_key: "882ee75bb4985f050e94781a6d6cee4a"
server_key_iv: "ddadd7448c82f6de2c89f96a8eedae54"
server_key_version: "encryptionkey@0"
I20260620 07:14:41.350535 27633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:41.356170 27633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:41.360117 27633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:41.360409 27633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:41.385475 27633 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34993
I20260620 07:14:41.385514 27699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34993 every 8 connection(s)
I20260620 07:14:41.386945 27633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:41.388020 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 27633
I20260620 07:14:41.388217 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:41.388756 6357 external_mini_cluster.cc:1428] Setting key a204cd719eb2752f24be52304746c460
I20260620 07:14:41.391480 27700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:41.396131 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:41.390509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:53495 (local address 127.0.0.1:34993)
0620 07:14:41.391233 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:41.391243 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:41.391282 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:14:41.392902 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:41.392921 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:41.393222 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:41.393590 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:41.394619 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:41.394624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:41.394636 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:41.394652 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:41.394722 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:41.394740 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:41.394982 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:41.395122 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 07:14:41.395293 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":308,"threads_started":1}
I20260620 07:14:41.399482 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:41.402458 27700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4: Bootstrap starting.
I20260620 07:14:41.406304 27700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:41.406985 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:41.404215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:50835 (local address 127.0.0.1:34993)
0620 07:14:41.404445 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:41.404455 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:41.404842 (+ 387us) server_negotiation.cc:407] Connection header received
0620 07:14:41.404965 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:41.404971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:41.405035 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:41.405155 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:41.406105 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:41.406112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:41.406117 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:41.406140 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:41.406200 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:41.406225 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:41.406408 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:41.406536 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 07:14:41.406613 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:14:41.407738 27700 log.cc:824] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:41.410614 27700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4: No bootstrap required, opened a new log
I20260620 07:14:41.418465 27700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331f718d980343b58eebc288bf39feb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34993 } }
I20260620 07:14:41.418807 27700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:41.418885 27700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 331f718d980343b58eebc288bf39feb4, State: Initialized, Role: FOLLOWER
I20260620 07:14:41.419488 27700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331f718d980343b58eebc288bf39feb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34993 } }
I20260620 07:14:41.419801 27700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:41.419975 27700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:41.420200 27700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:41.422509 27700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331f718d980343b58eebc288bf39feb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34993 } }
I20260620 07:14:41.423022 27700 leader_election.cc:302] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 331f718d980343b58eebc288bf39feb4; no voters:
I20260620 07:14:41.423623 27700 leader_election.cc:288] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:41.423719 27710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:41.424515 27710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [term 1 LEADER]: Becoming Leader. State: Replica: 331f718d980343b58eebc288bf39feb4, State: Running, Role: LEADER
I20260620 07:14:41.425029 27710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331f718d980343b58eebc288bf39feb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34993 } }
I20260620 07:14:41.425824 27700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:41.427793 27712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 331f718d980343b58eebc288bf39feb4. Latest consensus state: current_term: 1 leader_uuid: "331f718d980343b58eebc288bf39feb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331f718d980343b58eebc288bf39feb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34993 } } }
I20260620 07:14:41.428155 27712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:41.427973 27711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "331f718d980343b58eebc288bf39feb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331f718d980343b58eebc288bf39feb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34993 } } }
I20260620 07:14:41.428395 27711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:41.431113 27715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:41.436123 27715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:41.445708 27715 catalog_manager.cc:1347] Generated new cluster ID: fdcb59f9480d4dc1ac7508400756daf5
I20260620 07:14:41.445806 27715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:41.466012 27715 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:41.466475 27715 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:41.474656 27715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 331f718d980343b58eebc288bf39feb4: Generated new TSK 0
I20260620 07:14:41.475618 27715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:41.482527 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 27633
W20260620 07:14:41.483510 27695 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.218:53495
W20260620 07:14:41.484196 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 27633: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:14:41.484339 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 27633
2026-06-20T07:14:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1844 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:41 dist-test-slave-4n6h krb5kdc[27731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:41 dist-test-slave-4n6h krb5kdc[27731](info): set up 2 sockets
Jun 20 07:14:41 dist-test-slave-4n6h krb5kdc[27731](info): commencing operation
krb5kdc: starting...
W20260620 07:14:42.586791 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:42 dist-test-slave-4n6h krb5kdc[27731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:14:42.829200 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:43.146555 27744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:43.146816 27744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:43.147001 27744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:43.154377 27744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:43.154523 27744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:43.154631 27744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:43.154748 27744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:43.154831 27744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:43.164793 27744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34477
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:43.166019 27744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:43.167372 27744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:43.179795 27749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:43.179800 27752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:43.179816 27750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:43.180660 27744 server_base.cc:1034] running on GCE node
I20260620 07:14:43.181459 27744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:43.182842 27744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:43.184052 27744 hybrid_clock.cc:648] HybridClock initialized: now 1781939683183994 us; error 50 us; skew 500 ppm
I20260620 07:14:43.184422 27744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:43 dist-test-slave-4n6h krb5kdc[27731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:43.187557 27744 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:14:43.188975 27744 webserver.cc:460] Webserver started at http://127.0.0.1:39221/ using document root <none> and password file <none>
I20260620 07:14:43.189628 27744 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:43.189756 27744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:43.190048 27744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:43.192912 27744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "fd271af81024435996479640b76123fc"
format_stamp: "Formatted at 2026-06-20 07:14:43 on dist-test-slave-4n6h"
server_key: "16f0b3b1eea9da2e751fcab8611f4aba"
server_key_iv: "180ca4490013621dd2ed19bb9f11e205"
server_key_version: "encryptionkey@0"
I20260620 07:14:43.193643 27744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "fd271af81024435996479640b76123fc"
format_stamp: "Formatted at 2026-06-20 07:14:43 on dist-test-slave-4n6h"
server_key: "16f0b3b1eea9da2e751fcab8611f4aba"
server_key_iv: "180ca4490013621dd2ed19bb9f11e205"
server_key_version: "encryptionkey@0"
I20260620 07:14:43.198539 27744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:14:43.202075 27759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:43.203486 27744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:43.203622 27744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "fd271af81024435996479640b76123fc"
format_stamp: "Formatted at 2026-06-20 07:14:43 on dist-test-slave-4n6h"
server_key: "16f0b3b1eea9da2e751fcab8611f4aba"
server_key_iv: "180ca4490013621dd2ed19bb9f11e205"
server_key_version: "encryptionkey@0"
I20260620 07:14:43.203779 27744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:43.217985 27744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:43.222344 27744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:43.222575 27744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:43.240271 27744 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34477
I20260620 07:14:43.240309 27810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34477 every 8 connection(s)
I20260620 07:14:43.241647 27744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:43.245150 27811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:43.251434 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 27744
I20260620 07:14:43.251637 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:43.252074 6357 external_mini_cluster.cc:1428] Setting key 3cda999bc483f0045f35e0924b356090
I20260620 07:14:43.255689 27811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc: Bootstrap starting.
I20260620 07:14:43.259383 27811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:43.260764 27811 log.cc:824] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:43.263480 27811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc: No bootstrap required, opened a new log
Jun 20 07:14:43 dist-test-slave-4n6h krb5kdc[27731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:14:43.274163 27811 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd271af81024435996479640b76123fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34477 } }
I20260620 07:14:43.274616 27811 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:43.274740 27811 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd271af81024435996479640b76123fc, State: Initialized, Role: FOLLOWER
I20260620 07:14:43.275430 27811 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd271af81024435996479640b76123fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34477 } }
I20260620 07:14:43.275722 27811 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:43.275866 27811 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:43.276070 27811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:43.277190 27814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:43.253892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:59255 (local address 127.0.0.1:34477)
0620 07:14:43.254664 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:43.254679 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:43.254727 (+ 48us) server_negotiation.cc:407] Connection header received
0620 07:14:43.257216 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:43.257250 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:43.257786 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:43.258216 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:43.259548 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:43.261574 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:43.262917 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:43.263549 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:43.263724 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:43.268354 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:43.268366 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:43.268388 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:43.268435 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:43.272387 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:43.273201 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:43.273209 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:43.273217 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:43.273428 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:43.273839 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:43.273843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:43.273846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:43.274546 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:43.274901 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:43.275406 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 07:14:43.275788 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":338,"threads_started":1}
I20260620 07:14:43.279167 27811 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd271af81024435996479640b76123fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34477 } }
I20260620 07:14:43.279732 27811 leader_election.cc:302] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd271af81024435996479640b76123fc; no voters:
I20260620 07:14:43.280401 27811 leader_election.cc:288] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:43.280515 27816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:43.281213 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:43.282454 27811 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:43.287285 27816 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [term 1 LEADER]: Becoming Leader. State: Replica: fd271af81024435996479640b76123fc, State: Running, Role: LEADER
I20260620 07:14:43.287739 27816 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd271af81024435996479640b76123fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34477 } }
I20260620 07:14:43.290021 27817 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd271af81024435996479640b76123fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd271af81024435996479640b76123fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34477 } } }
I20260620 07:14:43.290434 27817 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:43.298851 27824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:43.298857 27818 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd271af81024435996479640b76123fc. Latest consensus state: current_term: 1 leader_uuid: "fd271af81024435996479640b76123fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd271af81024435996479640b76123fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34477 } } }
I20260620 07:14:43.299180 27818 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:43.304823 27824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:43.319458 27814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:43.300582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:48221 (local address 127.0.0.1:34477)
0620 07:14:43.300762 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:43.300770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:43.301741 (+ 971us) server_negotiation.cc:407] Connection header received
0620 07:14:43.301908 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:43.301913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:43.301996 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:43.302114 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:43.304144 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:43.305347 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:43.307299 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:43.308035 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:43.308128 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:43.311741 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:43.311748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:43.311753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:43.311789 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:43.315689 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:43.316692 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:43.316699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:43.316705 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:43.316860 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:43.317443 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:43.317450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:43.317453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:43.317852 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:43.318111 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:43.318873 (+ 762us) server_negotiation.cc:299] Negotiation successful
0620 07:14:43.319047 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:14:43.325158 27824 catalog_manager.cc:1347] Generated new cluster ID: 1d7da4b242ec435eaa59e41077bc1d5c
I20260620 07:14:43.325290 27824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:43.334863 27824 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:43.336544 27824 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:43.347749 27824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd271af81024435996479640b76123fc: Generated new TSK 0
I20260620 07:14:43.348660 27824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:43.355871 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 27744
I20260620 07:14:43.357905 27806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:43.445818 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 27744
W20260620 07:14:43.462549 27729 connection.cc:535] client connection to 127.0.0.1:34477 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34477): Connection reset by peer (error 104)
2026-06-20T07:14:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1961 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:43 dist-test-slave-4n6h krb5kdc[27844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:43 dist-test-slave-4n6h krb5kdc[27844](info): set up 2 sockets
Jun 20 07:14:43 dist-test-slave-4n6h krb5kdc[27844](info): commencing operation
krb5kdc: starting...
W20260620 07:14:44.566704 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:44 dist-test-slave-4n6h krb5kdc[27844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:14:44.811501 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:45.133931 27857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:45.134186 27857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:45.134378 27857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:45.141467 27857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:45.141572 27857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:45.141657 27857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:45.141757 27857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:45.141846 27857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:45.151595 27857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:45.152819 27857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:45.154075 27857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:45.166119 27862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:45.166213 27863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:45.166163 27865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:45.167285 27857 server_base.cc:1034] running on GCE node
I20260620 07:14:45.168058 27857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:45.169394 27857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:45.170646 27857 hybrid_clock.cc:648] HybridClock initialized: now 1781939685170598 us; error 58 us; skew 500 ppm
I20260620 07:14:45.170981 27857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:45 dist-test-slave-4n6h krb5kdc[27844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:45.177374 27857 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:14:45.178936 27857 webserver.cc:460] Webserver started at http://127.0.0.1:40909/ using document root <none> and password file <none>
I20260620 07:14:45.179543 27857 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:45.179649 27857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:45.180121 27857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:45.183982 27857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "e7ce2676ad3745a09cb5a1f93587e6f2"
format_stamp: "Formatted at 2026-06-20 07:14:45 on dist-test-slave-4n6h"
server_key: "3d0c0176e3b17489cc0f793fad05c3c7"
server_key_iv: "7651deab92e365b070d178a32592a0cf"
server_key_version: "encryptionkey@0"
I20260620 07:14:45.184688 27857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "e7ce2676ad3745a09cb5a1f93587e6f2"
format_stamp: "Formatted at 2026-06-20 07:14:45 on dist-test-slave-4n6h"
server_key: "3d0c0176e3b17489cc0f793fad05c3c7"
server_key_iv: "7651deab92e365b070d178a32592a0cf"
server_key_version: "encryptionkey@0"
I20260620 07:14:45.189654 27857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 07:14:45.192950 27872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:45.194232 27857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:14:45.194505 27857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "e7ce2676ad3745a09cb5a1f93587e6f2"
format_stamp: "Formatted at 2026-06-20 07:14:45 on dist-test-slave-4n6h"
server_key: "3d0c0176e3b17489cc0f793fad05c3c7"
server_key_iv: "7651deab92e365b070d178a32592a0cf"
server_key_version: "encryptionkey@0"
I20260620 07:14:45.194707 27857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:45.209857 27857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:45.213729 27857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:45.214012 27857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:45.233060 27857 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35641
I20260620 07:14:45.233080 27923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35641 every 8 connection(s)
I20260620 07:14:45.234572 27857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:45.236027 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 27857
I20260620 07:14:45.236217 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:45.236759 6357 external_mini_cluster.cc:1428] Setting key 17262b5cc99b5ea3e6255315872fe9ed
I20260620 07:14:45.239042 27924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:45.249819 27927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:45.238628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:43711 (local address 127.0.0.1:35641)
0620 07:14:45.239389 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:45.239400 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:45.239438 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:14:45.241023 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:45.241044 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:45.241426 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:45.241744 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:45.242367 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:45.243703 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:45.245489 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:45.246109 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:45.246227 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:45.247545 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:45.247587 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:45.247600 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:45.248174 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:45.248243 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:45.248274 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:45.248402 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:45.248752 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 07:14:45.248929 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":349,"threads_started":1}
I20260620 07:14:45.251220 27924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2: Bootstrap starting.
I20260620 07:14:45.254015 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:45.255158 27924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:45.256597 27924 log.cc:824] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:45.260131 27924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2: No bootstrap required, opened a new log
I20260620 07:14:45.271528 27927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:45.263100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:38981 (local address 127.0.0.1:35641)
0620 07:14:45.263283 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:45.263290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:45.264123 (+ 833us) server_negotiation.cc:407] Connection header received
0620 07:14:45.264239 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:45.264246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:45.264328 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:45.264458 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:45.265311 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:45.266716 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:45.268401 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:45.269301 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:45.269434 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:45.270698 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:45.270703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:45.270706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:45.270723 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:45.270759 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:45.270773 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:45.270898 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:45.271130 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 07:14:45.271188 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:14:45.272503 27924 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ce2676ad3745a09cb5a1f93587e6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } }
I20260620 07:14:45.272948 27924 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:45.273077 27924 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ce2676ad3745a09cb5a1f93587e6f2, State: Initialized, Role: FOLLOWER
I20260620 07:14:45.273696 27924 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ce2676ad3745a09cb5a1f93587e6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } }
I20260620 07:14:45.273938 27924 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:45.274070 27924 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:45.274284 27924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:45.277333 27924 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ce2676ad3745a09cb5a1f93587e6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } }
I20260620 07:14:45.277974 27924 leader_election.cc:302] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7ce2676ad3745a09cb5a1f93587e6f2; no voters:
I20260620 07:14:45.278726 27924 leader_election.cc:288] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:45.278887 27934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:45.280427 27934 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [term 1 LEADER]: Becoming Leader. State: Replica: e7ce2676ad3745a09cb5a1f93587e6f2, State: Running, Role: LEADER
I20260620 07:14:45.281077 27924 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:45.281004 27934 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ce2676ad3745a09cb5a1f93587e6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } }
I20260620 07:14:45.289485 27936 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7ce2676ad3745a09cb5a1f93587e6f2. Latest consensus state: current_term: 1 leader_uuid: "e7ce2676ad3745a09cb5a1f93587e6f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ce2676ad3745a09cb5a1f93587e6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } } }
I20260620 07:14:45.289863 27936 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:45.303699 27935 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7ce2676ad3745a09cb5a1f93587e6f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ce2676ad3745a09cb5a1f93587e6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35641 } } }
I20260620 07:14:45.304694 27935 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2 [sys.catalog]: This master's current role is: LEADER
W20260620 07:14:45.312685 27948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:14:45.312829 27948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:14:45.316058 27950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:45.317591 27950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:45.325909 27950 catalog_manager.cc:1347] Generated new cluster ID: 55bb9dd74fd84a2482fc48d73c99a918
I20260620 07:14:45.326005 27950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:45.354199 27950 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:45.356041 27950 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:45.362979 27950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7ce2676ad3745a09cb5a1f93587e6f2: Generated new TSK 0
I20260620 07:14:45.363986 27950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:45.373749 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 27857
W20260620 07:14:45.375322 27919 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.218:43711
W20260620 07:14:45.376134 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 27857: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:14:45.376296 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 27857
W20260620 07:14:45.390151 27842 connection.cc:535] client connection to 127.0.0.1:35641 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35641): Connection reset by peer (error 104)
2026-06-20T07:14:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1927 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:45 dist-test-slave-4n6h krb5kdc[27955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:45 dist-test-slave-4n6h krb5kdc[27955](info): set up 2 sockets
Jun 20 07:14:45 dist-test-slave-4n6h krb5kdc[27955](info): commencing operation
krb5kdc: starting...
W20260620 07:14:46.470781 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:46 dist-test-slave-4n6h krb5kdc[27955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:14:46.713416 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42237
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:47.044909 27968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:47.045245 27968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:47.045428 27968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:47.053439 27968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:47.053555 27968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:47.053678 27968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:47.053797 27968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:47.053866 27968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:47.064178 27968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42237
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:47.065526 27968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:47.066926 27968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:47.078351 27976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:47.078397 27973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:47.078444 27974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:47.079308 27968 server_base.cc:1034] running on GCE node
I20260620 07:14:47.079967 27968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:47.081190 27968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:47.082358 27968 hybrid_clock.cc:648] HybridClock initialized: now 1781939687082313 us; error 35 us; skew 500 ppm
I20260620 07:14:47.082752 27968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:47 dist-test-slave-4n6h krb5kdc[27955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:47.089653 27968 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:14:47.091176 27968 webserver.cc:460] Webserver started at http://127.0.0.1:40645/ using document root <none> and password file <none>
I20260620 07:14:47.091840 27968 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:47.091986 27968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:47.092290 27968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:47.095252 27968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "2903498840e14bf792f57960a8767729"
format_stamp: "Formatted at 2026-06-20 07:14:47 on dist-test-slave-4n6h"
server_key: "bf4eefde5d9298f614460ef779ad5105"
server_key_iv: "31e7d14befb29729d9ffd785f0365474"
server_key_version: "encryptionkey@0"
I20260620 07:14:47.096024 27968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "2903498840e14bf792f57960a8767729"
format_stamp: "Formatted at 2026-06-20 07:14:47 on dist-test-slave-4n6h"
server_key: "bf4eefde5d9298f614460ef779ad5105"
server_key_iv: "31e7d14befb29729d9ffd785f0365474"
server_key_version: "encryptionkey@0"
I20260620 07:14:47.101033 27968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 07:14:47.104382 27983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:47.105626 27968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 07:14:47.105800 27968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "2903498840e14bf792f57960a8767729"
format_stamp: "Formatted at 2026-06-20 07:14:47 on dist-test-slave-4n6h"
server_key: "bf4eefde5d9298f614460ef779ad5105"
server_key_iv: "31e7d14befb29729d9ffd785f0365474"
server_key_version: "encryptionkey@0"
I20260620 07:14:47.105993 27968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:47.111693 27968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:47.115489 27968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:47.115783 27968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:47.133795 27968 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42237
I20260620 07:14:47.133870 28034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42237 every 8 connection(s)
I20260620 07:14:47.135479 27968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:47.136966 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 27968
I20260620 07:14:47.137126 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:47.137578 6357 external_mini_cluster.cc:1428] Setting key 9564c5f477b8b2dc3e6c24dd53877b2f
I20260620 07:14:47.139719 28035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:47.144939 28038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:47.139272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:57211 (local address 127.0.0.1:42237)
0620 07:14:47.140051 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:47.140063 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:47.140108 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:14:47.141696 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:47.141715 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:47.142036 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:47.142429 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:47.143447 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:47.143454 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:47.143466 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:47.143479 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:47.143548 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:47.143566 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:47.143747 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:47.143862 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 07:14:47.144063 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":343,"threads_started":1}
I20260620 07:14:47.148367 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:47.151013 28035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729: Bootstrap starting.
I20260620 07:14:47.154529 28035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:47.155205 28038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:47.152674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.218:54125 (local address 127.0.0.1:42237)
0620 07:14:47.152869 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:47.152875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:47.153203 (+ 328us) server_negotiation.cc:407] Connection header received
0620 07:14:47.153367 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:47.153370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:47.153411 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:47.153478 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:47.154445 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:47.154449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:47.154452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:47.154467 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:47.154505 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:47.154522 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:47.154696 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:47.154834 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 07:14:47.154885 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:14:47.155797 28035 log.cc:824] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:47.158387 28035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729: No bootstrap required, opened a new log
I20260620 07:14:47.165315 28035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2903498840e14bf792f57960a8767729" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } }
I20260620 07:14:47.165686 28035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:47.165791 28035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2903498840e14bf792f57960a8767729, State: Initialized, Role: FOLLOWER
I20260620 07:14:47.166378 28035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2903498840e14bf792f57960a8767729" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } }
I20260620 07:14:47.166610 28035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:47.166760 28035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:47.166923 28035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:47.169082 28035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2903498840e14bf792f57960a8767729" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } }
I20260620 07:14:47.169554 28035 leader_election.cc:302] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2903498840e14bf792f57960a8767729; no voters:
I20260620 07:14:47.170109 28035 leader_election.cc:288] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:47.170274 28045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:47.170938 28045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [term 1 LEADER]: Becoming Leader. State: Replica: 2903498840e14bf792f57960a8767729, State: Running, Role: LEADER
I20260620 07:14:47.171527 28045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2903498840e14bf792f57960a8767729" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } }
I20260620 07:14:47.172410 28035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:47.174458 28046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2903498840e14bf792f57960a8767729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2903498840e14bf792f57960a8767729" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } } }
I20260620 07:14:47.174813 28046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:47.175523 28047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2903498840e14bf792f57960a8767729. Latest consensus state: current_term: 1 leader_uuid: "2903498840e14bf792f57960a8767729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2903498840e14bf792f57960a8767729" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42237 } } }
I20260620 07:14:47.175803 28047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:47.179312 28052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:47.184585 28052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:47.194317 28052 catalog_manager.cc:1347] Generated new cluster ID: 724833a19eb4498787018bb54388b7d5
I20260620 07:14:47.194434 28052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:47.206051 28052 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:47.206486 28052 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:47.213179 28052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2903498840e14bf792f57960a8767729: Generated new TSK 0
I20260620 07:14:47.213904 28052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:47.226120 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 27968
W20260620 07:14:47.227206 28030 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.218:57211
W20260620 07:14:47.227941 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 27968: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:14:47.228080 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 27968
2026-06-20T07:14:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1851 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:47 dist-test-slave-4n6h krb5kdc[28066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:47 dist-test-slave-4n6h krb5kdc[28066](info): set up 2 sockets
Jun 20 07:14:47 dist-test-slave-4n6h krb5kdc[28066](info): commencing operation
krb5kdc: starting...
W20260620 07:14:48.334622 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:48 dist-test-slave-4n6h krb5kdc[28066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:14:48.575470 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:36243
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:37793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:36243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:48.893898 28079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:48.894166 28079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:48.894352 28079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:48.901528 28079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:48.901630 28079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:48.901743 28079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:48.901846 28079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:48.901927 28079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:48.911669 28079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:37793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:36243
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:36243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:48.912894 28079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:48.914206 28079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:48.926072 28084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:48.926107 28085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:48.926162 28087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:48.926887 28079 server_base.cc:1034] running on GCE node
I20260620 07:14:48.927599 28079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:48.928854 28079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:48.930047 28079 hybrid_clock.cc:648] HybridClock initialized: now 1781939688930003 us; error 52 us; skew 500 ppm
I20260620 07:14:48.930444 28079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:48 dist-test-slave-4n6h krb5kdc[28066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939688, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:48.933476 28079 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:14:48.934897 28079 webserver.cc:460] Webserver started at http://127.6.53.126:40805/ using document root <none> and password file <none>
I20260620 07:14:48.935541 28079 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:48.935689 28079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:48.935981 28079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:48.938890 28079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "b29eabac9be04443ac5fb0901b236266"
format_stamp: "Formatted at 2026-06-20 07:14:48 on dist-test-slave-4n6h"
server_key: "65806bde1d33133b78b4618ff7afcb04"
server_key_iv: "76842533fa60ab67b946ce60d89d2e38"
server_key_version: "encryptionkey@0"
I20260620 07:14:48.939661 28079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "b29eabac9be04443ac5fb0901b236266"
format_stamp: "Formatted at 2026-06-20 07:14:48 on dist-test-slave-4n6h"
server_key: "65806bde1d33133b78b4618ff7afcb04"
server_key_iv: "76842533fa60ab67b946ce60d89d2e38"
server_key_version: "encryptionkey@0"
I20260620 07:14:48.944428 28079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:14:48.947710 28094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:48.948918 28079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:14:48.949093 28079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "b29eabac9be04443ac5fb0901b236266"
format_stamp: "Formatted at 2026-06-20 07:14:48 on dist-test-slave-4n6h"
server_key: "65806bde1d33133b78b4618ff7afcb04"
server_key_iv: "76842533fa60ab67b946ce60d89d2e38"
server_key_version: "encryptionkey@0"
I20260620 07:14:48.949294 28079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:48.962057 28079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:48.966192 28079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:48.966496 28079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:48.984877 28079 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:36243
I20260620 07:14:48.984898 28145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:36243 every 8 connection(s)
I20260620 07:14:48.986179 28079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:48.987542 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 28079
I20260620 07:14:48.987727 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:48.988253 6357 external_mini_cluster.cc:1428] Setting key 4faa41f437193911529e4ba5dd85e12e
I20260620 07:14:48.990566 28146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:49.000639 28146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266: Bootstrap starting.
Jun 20 07:14:48 dist-test-slave-4n6h krb5kdc[28066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:14:49.004098 28146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:49.005437 28146 log.cc:824] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:49.008831 28146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266: No bootstrap required, opened a new log
I20260620 07:14:49.011032 28149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:48.989972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40925 (local address 127.6.53.126:36243)
0620 07:14:48.990769 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:48.990780 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:48.990817 (+ 37us) server_negotiation.cc:407] Connection header received
0620 07:14:48.992398 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:48.992418 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:48.992773 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:48.993151 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:48.994570 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:48.995848 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:48.997114 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:48.997793 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:48.997988 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:49.002515 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:49.002523 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:49.002536 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:49.002583 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:49.006373 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:49.007358 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:49.007366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:49.007373 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:49.007566 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:49.007995 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:49.008000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:49.008003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:49.008660 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:49.008977 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:49.009412 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 07:14:49.009751 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":337,"threads_started":1}
I20260620 07:14:49.014902 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:49.017782 28146 raft_consensus.cc:348] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29eabac9be04443ac5fb0901b236266" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36243 } }
I20260620 07:14:49.018136 28146 raft_consensus.cc:374] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:49.018293 28146 raft_consensus.cc:729] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b29eabac9be04443ac5fb0901b236266, State: Initialized, Role: FOLLOWER
I20260620 07:14:49.018926 28146 consensus_queue.cc:260] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29eabac9be04443ac5fb0901b236266" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36243 } }
I20260620 07:14:49.019228 28146 raft_consensus.cc:388] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:49.019376 28146 raft_consensus.cc:482] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:49.019587 28146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:49.021746 28146 raft_consensus.cc:504] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29eabac9be04443ac5fb0901b236266" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36243 } }
I20260620 07:14:49.022307 28146 leader_election.cc:302] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b29eabac9be04443ac5fb0901b236266; no voters:
I20260620 07:14:49.022949 28146 leader_election.cc:288] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:49.023051 28156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:49.025358 28146 sys_catalog.cc:564] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:49.026542 28156 raft_consensus.cc:686] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [term 1 LEADER]: Becoming Leader. State: Replica: b29eabac9be04443ac5fb0901b236266, State: Running, Role: LEADER
I20260620 07:14:49.027173 28156 consensus_queue.cc:237] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29eabac9be04443ac5fb0901b236266" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36243 } }
I20260620 07:14:49.031066 28157 sys_catalog.cc:455] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b29eabac9be04443ac5fb0901b236266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29eabac9be04443ac5fb0901b236266" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36243 } } }
I20260620 07:14:49.031513 28157 sys_catalog.cc:458] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:49.036891 28149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:49.019111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57503 (local address 127.6.53.126:36243)
0620 07:14:49.019261 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:49.019265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:49.019930 (+ 665us) server_negotiation.cc:407] Connection header received
0620 07:14:49.020130 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:49.020135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:49.020204 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:49.020322 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:49.021611 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:49.022586 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:49.024207 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:49.024668 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:49.024737 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:49.028017 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:49.028022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:49.028025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:49.028054 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:49.031756 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:49.033002 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:49.033011 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:49.033017 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:49.033250 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:49.033951 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:49.033958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:49.033964 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:49.034458 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:49.035938 (+ 1480us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:49.036165 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 07:14:49.036390 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:14:49.039361 28158 sys_catalog.cc:455] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b29eabac9be04443ac5fb0901b236266. Latest consensus state: current_term: 1 leader_uuid: "b29eabac9be04443ac5fb0901b236266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b29eabac9be04443ac5fb0901b236266" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36243 } } }
I20260620 07:14:49.039683 28158 sys_catalog.cc:458] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266 [sys.catalog]: This master's current role is: LEADER
W20260620 07:14:49.044724 28172 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:14:49.044857 28172 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:14:49.045014 28164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:49.046706 28164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:49.055120 28164 catalog_manager.cc:1347] Generated new cluster ID: aa6c1779d1574e87a75936e7124782a3
I20260620 07:14:49.055264 28164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:49.069968 28164 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:49.071691 28164 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:49.093142 28164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b29eabac9be04443ac5fb0901b236266: Generated new TSK 0
I20260620 07:14:49.094132 28164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:49.107342 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 28079
I20260620 07:14:49.108729 28141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:49.201001 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 28079
W20260620 07:14:49.218030 28064 connection.cc:535] client connection to 127.6.53.126:36243 recv error: Network error: failed to read from TLS socket (remote: 127.6.53.126:36243): Connection reset by peer (error 104)
2026-06-20T07:14:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:49 dist-test-slave-4n6h krb5kdc[28178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:49 dist-test-slave-4n6h krb5kdc[28178](info): set up 2 sockets
Jun 20 07:14:49 dist-test-slave-4n6h krb5kdc[28178](info): commencing operation
krb5kdc: starting...
W20260620 07:14:50.298622 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:50 dist-test-slave-4n6h krb5kdc[28178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:14:50.531545 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:37887
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:36533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:37887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:50.849565 28191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:50.849836 28191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:50.850036 28191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:50.857230 28191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:50.857342 28191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:50.857455 28191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:50.857571 28191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:50.857650 28191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:50.867528 28191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:36533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:37887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:37887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:50.868770 28191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:50.869990 28191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:50.881624 28196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:50.881654 28197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:50.881707 28199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:50.882841 28191 server_base.cc:1034] running on GCE node
I20260620 07:14:50.883555 28191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:50.884811 28191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:50.886042 28191 hybrid_clock.cc:648] HybridClock initialized: now 1781939690886002 us; error 48 us; skew 500 ppm
I20260620 07:14:50.886469 28191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:50 dist-test-slave-4n6h krb5kdc[28178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939690, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:50.889472 28191 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:14:50.890847 28191 webserver.cc:460] Webserver started at http://127.6.53.126:44163/ using document root <none> and password file <none>
I20260620 07:14:50.891481 28191 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:50.891579 28191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:50.891809 28191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:50.894675 28191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "6dec364573a247abaf8eb19e5a02f2d1"
format_stamp: "Formatted at 2026-06-20 07:14:50 on dist-test-slave-4n6h"
server_key: "afb5c9ee2963e83ab9b7fdce8604070f"
server_key_iv: "102247210f70688e157d1000514d5893"
server_key_version: "encryptionkey@0"
I20260620 07:14:50.895362 28191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "6dec364573a247abaf8eb19e5a02f2d1"
format_stamp: "Formatted at 2026-06-20 07:14:50 on dist-test-slave-4n6h"
server_key: "afb5c9ee2963e83ab9b7fdce8604070f"
server_key_iv: "102247210f70688e157d1000514d5893"
server_key_version: "encryptionkey@0"
I20260620 07:14:50.900218 28191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:14:50.903527 28206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:50.904853 28191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:14:50.904994 28191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "6dec364573a247abaf8eb19e5a02f2d1"
format_stamp: "Formatted at 2026-06-20 07:14:50 on dist-test-slave-4n6h"
server_key: "afb5c9ee2963e83ab9b7fdce8604070f"
server_key_iv: "102247210f70688e157d1000514d5893"
server_key_version: "encryptionkey@0"
I20260620 07:14:50.905169 28191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:50.925913 28191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:50.929994 28191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:50.930225 28191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:50.948470 28191 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:37887
I20260620 07:14:50.948503 28257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:37887 every 8 connection(s)
I20260620 07:14:50.949923 28191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:50.952622 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 28191
I20260620 07:14:50.952802 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:50.953316 6357 external_mini_cluster.cc:1428] Setting key 859fe3c40349c210939dd7e4ac2e2d25
I20260620 07:14:50.953941 28258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:50.965898 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:50.955105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44441 (local address 127.6.53.126:37887)
0620 07:14:50.955816 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:50.955826 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:50.955864 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:14:50.957362 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:50.957381 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:50.957730 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:50.958153 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:50.958682 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:50.960001 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:50.961327 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:50.961964 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:50.962159 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:50.963460 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:50.963468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:50.963484 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:50.964069 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:50.964140 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:50.964168 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:50.964330 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:50.964605 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 07:14:50.964757 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":321,"threads_started":1}
I20260620 07:14:50.966168 28258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1: Bootstrap starting.
I20260620 07:14:50.969236 28258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:50.969570 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:50.970307 28258 log.cc:824] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:50.973069 28258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1: No bootstrap required, opened a new log
I20260620 07:14:50.980124 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:50.973804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46945 (local address 127.6.53.126:37887)
0620 07:14:50.974136 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:50.974145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:50.974704 (+ 559us) server_negotiation.cc:407] Connection header received
0620 07:14:50.974915 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:50.974919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:50.974990 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:50.975100 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:50.975608 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:50.976438 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:50.977661 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:50.978108 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:50.978261 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:50.979411 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:50.979419 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:50.979422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:50.979440 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:50.979487 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:50.979501 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:50.979645 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:50.979817 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 07:14:50.979869 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 07:14:50.981084 28258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dec364573a247abaf8eb19e5a02f2d1" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 37887 } }
I20260620 07:14:50.981501 28258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:50.981616 28258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dec364573a247abaf8eb19e5a02f2d1, State: Initialized, Role: FOLLOWER
I20260620 07:14:50.982059 28258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dec364573a247abaf8eb19e5a02f2d1" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 37887 } }
I20260620 07:14:50.982313 28258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:50.982441 28258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:50.982584 28258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:50.984594 28258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dec364573a247abaf8eb19e5a02f2d1" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 37887 } }
I20260620 07:14:50.985070 28258 leader_election.cc:302] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dec364573a247abaf8eb19e5a02f2d1; no voters:
I20260620 07:14:50.985554 28258 leader_election.cc:288] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:50.985731 28268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:50.986430 28268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [term 1 LEADER]: Becoming Leader. State: Replica: 6dec364573a247abaf8eb19e5a02f2d1, State: Running, Role: LEADER
I20260620 07:14:50.986989 28268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dec364573a247abaf8eb19e5a02f2d1" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 37887 } }
I20260620 07:14:50.988695 28258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:50.989843 28270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dec364573a247abaf8eb19e5a02f2d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dec364573a247abaf8eb19e5a02f2d1" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 37887 } } }
I20260620 07:14:50.989871 28269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dec364573a247abaf8eb19e5a02f2d1. Latest consensus state: current_term: 1 leader_uuid: "6dec364573a247abaf8eb19e5a02f2d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dec364573a247abaf8eb19e5a02f2d1" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 37887 } } }
I20260620 07:14:50.990278 28270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:50.990288 28269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:50.992205 28273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:50.998518 28273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:51.008313 28273 catalog_manager.cc:1347] Generated new cluster ID: c7eb2b6fd0284297b0bab95a4b32b379
I20260620 07:14:51.008435 28273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:51.052026 28273 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:51.053848 28273 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:51.061081 28273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dec364573a247abaf8eb19e5a02f2d1: Generated new TSK 0
I20260620 07:14:51.061848 28273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:51.075783 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 28191
W20260620 07:14:51.076817 28253 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44441
W20260620 07:14:51.077569 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 28191: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:14:51.077713 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 28191
W20260620 07:14:51.094401 28176 connection.cc:535] client connection to 127.6.53.126:37887 recv error: Network error: failed to read from TLS socket (remote: 127.6.53.126:37887): Connection reset by peer (error 104)
2026-06-20T07:14:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1876 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:51 dist-test-slave-4n6h krb5kdc[28289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:51 dist-test-slave-4n6h krb5kdc[28289](info): set up 2 sockets
Jun 20 07:14:51 dist-test-slave-4n6h krb5kdc[28289](info): commencing operation
krb5kdc: starting...
W20260620 07:14:52.182947 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:52 dist-test-slave-4n6h krb5kdc[28289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:14:52.429775 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:39057
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:39057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:52.751444 28302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:52.751718 28302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:52.751811 28302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:52.758942 28302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:52.759050 28302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:52.759096 28302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:52.759155 28302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:52.759198 28302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:52.769007 28302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:39057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:39057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:52.770172 28302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:52.771492 28302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:52.783299 28307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:52.783358 28310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:52.783325 28308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:52.784525 28302 server_base.cc:1034] running on GCE node
I20260620 07:14:52.785236 28302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:52.786643 28302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:52.787847 28302 hybrid_clock.cc:648] HybridClock initialized: now 1781939692787811 us; error 34 us; skew 500 ppm
I20260620 07:14:52.788174 28302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:52 dist-test-slave-4n6h krb5kdc[28289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939692, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:52.796041 28302 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:14:52.797546 28302 webserver.cc:460] Webserver started at http://127.6.53.126:37175/ using document root <none> and password file <none>
I20260620 07:14:52.798221 28302 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:52.798432 28302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:52.798759 28302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:52.801645 28302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "e28019b1698345aa844eb21e5f1b50f5"
format_stamp: "Formatted at 2026-06-20 07:14:52 on dist-test-slave-4n6h"
server_key: "095f6b297d92afc7468ce5a50f025945"
server_key_iv: "85d1d59802fa2f182400fa65554173d5"
server_key_version: "encryptionkey@0"
I20260620 07:14:52.802470 28302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "e28019b1698345aa844eb21e5f1b50f5"
format_stamp: "Formatted at 2026-06-20 07:14:52 on dist-test-slave-4n6h"
server_key: "095f6b297d92afc7468ce5a50f025945"
server_key_iv: "85d1d59802fa2f182400fa65554173d5"
server_key_version: "encryptionkey@0"
I20260620 07:14:52.807399 28302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:14:52.810662 28317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:52.811872 28302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 07:14:52.812043 28302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "e28019b1698345aa844eb21e5f1b50f5"
format_stamp: "Formatted at 2026-06-20 07:14:52 on dist-test-slave-4n6h"
server_key: "095f6b297d92afc7468ce5a50f025945"
server_key_iv: "85d1d59802fa2f182400fa65554173d5"
server_key_version: "encryptionkey@0"
I20260620 07:14:52.812219 28302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:52.817808 28302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:52.821511 28302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:52.821787 28302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:52.840170 28302 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:39057
I20260620 07:14:52.840240 28368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:39057 every 8 connection(s)
I20260620 07:14:52.841480 28302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:52.842850 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 28302
I20260620 07:14:52.843025 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:52.843576 6357 external_mini_cluster.cc:1428] Setting key 2375410357b885ed6ca6cf8f2528736f
I20260620 07:14:52.845839 28369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:52.851076 28372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:52.845306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59603 (local address 127.6.53.126:39057)
0620 07:14:52.846103 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:52.846117 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:52.846155 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:14:52.847779 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:52.847799 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:52.848129 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:52.848443 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:52.849495 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:52.849501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:52.849516 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:52.849536 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:52.849600 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:52.849622 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:52.849783 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:52.849914 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 07:14:52.850151 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":357,"threads_started":1}
I20260620 07:14:52.854447 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:52.858047 28369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5: Bootstrap starting.
I20260620 07:14:52.862107 28369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:52.862130 28372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:52.859813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46297 (local address 127.6.53.126:39057)
0620 07:14:52.859972 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:52.859976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:52.860256 (+ 280us) server_negotiation.cc:407] Connection header received
0620 07:14:52.860420 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:52.860423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:52.860469 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:52.860547 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:52.861462 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:52.861466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:52.861469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:52.861488 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:52.861523 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:52.861537 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:52.861631 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:52.861764 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 07:14:52.861806 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:14:52.863224 28369 log.cc:824] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:52.865696 28369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5: No bootstrap required, opened a new log
I20260620 07:14:52.872721 28369 raft_consensus.cc:348] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28019b1698345aa844eb21e5f1b50f5" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39057 } }
I20260620 07:14:52.873091 28369 raft_consensus.cc:374] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:52.873198 28369 raft_consensus.cc:729] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28019b1698345aa844eb21e5f1b50f5, State: Initialized, Role: FOLLOWER
I20260620 07:14:52.873725 28369 consensus_queue.cc:260] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28019b1698345aa844eb21e5f1b50f5" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39057 } }
I20260620 07:14:52.873956 28369 raft_consensus.cc:388] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:52.874101 28369 raft_consensus.cc:482] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:52.874281 28369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:52.876464 28369 raft_consensus.cc:504] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28019b1698345aa844eb21e5f1b50f5" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39057 } }
I20260620 07:14:52.876928 28369 leader_election.cc:302] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e28019b1698345aa844eb21e5f1b50f5; no voters:
I20260620 07:14:52.877476 28369 leader_election.cc:288] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:52.877645 28379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:52.878314 28379 raft_consensus.cc:686] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [term 1 LEADER]: Becoming Leader. State: Replica: e28019b1698345aa844eb21e5f1b50f5, State: Running, Role: LEADER
I20260620 07:14:52.878856 28379 consensus_queue.cc:237] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28019b1698345aa844eb21e5f1b50f5" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39057 } }
I20260620 07:14:52.879446 28369 sys_catalog.cc:564] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:52.881651 28381 sys_catalog.cc:455] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e28019b1698345aa844eb21e5f1b50f5. Latest consensus state: current_term: 1 leader_uuid: "e28019b1698345aa844eb21e5f1b50f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28019b1698345aa844eb21e5f1b50f5" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39057 } } }
I20260620 07:14:52.882085 28381 sys_catalog.cc:458] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:52.881670 28380 sys_catalog.cc:455] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e28019b1698345aa844eb21e5f1b50f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28019b1698345aa844eb21e5f1b50f5" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 39057 } } }
I20260620 07:14:52.882478 28380 sys_catalog.cc:458] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:52.883920 28384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:52.888726 28384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:52.897334 28384 catalog_manager.cc:1347] Generated new cluster ID: 25c3a67e524a4e1fabc066d81847b8a5
I20260620 07:14:52.897433 28384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:52.913720 28384 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:52.914155 28384 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:52.925906 28384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e28019b1698345aa844eb21e5f1b50f5: Generated new TSK 0
I20260620 07:14:52.926766 28384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:52.935101 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 28302
W20260620 07:14:52.936141 28364 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59603
W20260620 07:14:52.936770 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 28302: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:14:52.936913 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 28302
2026-06-20T07:14:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1859 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:53 dist-test-slave-4n6h krb5kdc[28400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:53 dist-test-slave-4n6h krb5kdc[28400](info): set up 2 sockets
Jun 20 07:14:53 dist-test-slave-4n6h krb5kdc[28400](info): commencing operation
krb5kdc: starting...
W20260620 07:14:54.086824 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.010s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:54 dist-test-slave-4n6h krb5kdc[28400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:14:54.326646 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:36011
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:45569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:36011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:54.676354 28413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:54.676607 28413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:54.676765 28413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:54.683822 28413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:54.683956 28413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:54.684103 28413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:54.684211 28413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:54.684294 28413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:54.693794 28413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:45569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:36011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:36011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:54.694996 28413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:54.696192 28413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:54.707213 28418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:54.707278 28419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:54.707368 28421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:54.708325 28413 server_base.cc:1034] running on GCE node
I20260620 07:14:54.709043 28413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:54.710319 28413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:54.711539 28413 hybrid_clock.cc:648] HybridClock initialized: now 1781939694711507 us; error 30 us; skew 500 ppm
I20260620 07:14:54.711843 28413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:54 dist-test-slave-4n6h krb5kdc[28400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939694, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:54.715050 28413 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:14:54.716545 28413 webserver.cc:460] Webserver started at http://127.6.53.126:43881/ using document root <none> and password file <none>
I20260620 07:14:54.717200 28413 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:54.717347 28413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:54.717646 28413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:54.720587 28413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "2b28075a4ad5460babfe98e71787cd40"
format_stamp: "Formatted at 2026-06-20 07:14:54 on dist-test-slave-4n6h"
server_key: "2d742ac6d5b7c98a8dd88edff89dab57"
server_key_iv: "7fcc4542e63522d25a27677cf1081bbc"
server_key_version: "encryptionkey@0"
I20260620 07:14:54.721366 28413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "2b28075a4ad5460babfe98e71787cd40"
format_stamp: "Formatted at 2026-06-20 07:14:54 on dist-test-slave-4n6h"
server_key: "2d742ac6d5b7c98a8dd88edff89dab57"
server_key_iv: "7fcc4542e63522d25a27677cf1081bbc"
server_key_version: "encryptionkey@0"
I20260620 07:14:54.726150 28413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 07:14:54.729367 28428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:54.730641 28413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:54.730820 28413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "2b28075a4ad5460babfe98e71787cd40"
format_stamp: "Formatted at 2026-06-20 07:14:54 on dist-test-slave-4n6h"
server_key: "2d742ac6d5b7c98a8dd88edff89dab57"
server_key_iv: "7fcc4542e63522d25a27677cf1081bbc"
server_key_version: "encryptionkey@0"
I20260620 07:14:54.730979 28413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:54.745126 28413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:54.748811 28413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:54.749081 28413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:54.768023 28413 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:36011
I20260620 07:14:54.768071 28479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:36011 every 8 connection(s)
I20260620 07:14:54.769358 28413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:54.772294 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 28413
I20260620 07:14:54.772509 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:54.773000 6357 external_mini_cluster.cc:1428] Setting key 075e00ecff9de3a0a7f2a4f5d2b7817d
I20260620 07:14:54.773316 28480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:54.788866 28480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40: Bootstrap starting.
Jun 20 07:14:54 dist-test-slave-4n6h krb5kdc[28400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:14:54.793118 28480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:54.794585 28480 log.cc:824] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:54.798026 28480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40: No bootstrap required, opened a new log
I20260620 07:14:54.799110 28483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:54.774771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33243 (local address 127.6.53.126:36011)
0620 07:14:54.775580 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:54.775591 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:54.775632 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:14:54.778045 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:54.778065 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:54.778528 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:54.778868 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:54.780771 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:54.782153 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:54.784084 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:54.784834 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:54.784964 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:54.790032 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:54.790042 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:54.790061 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:54.790105 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:54.794428 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:54.795190 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:54.795204 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:54.795216 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:54.795424 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:54.795899 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:54.795904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:54.795907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:54.796641 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:54.796951 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:54.797256 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 07:14:54.797674 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":402,"threads_started":1}
I20260620 07:14:54.802992 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:54.807272 28480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b28075a4ad5460babfe98e71787cd40" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36011 } }
I20260620 07:14:54.807698 28480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:54.807852 28480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b28075a4ad5460babfe98e71787cd40, State: Initialized, Role: FOLLOWER
I20260620 07:14:54.808511 28480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b28075a4ad5460babfe98e71787cd40" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36011 } }
I20260620 07:14:54.808775 28480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:54.808938 28480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:54.809132 28480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:54.811595 28480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b28075a4ad5460babfe98e71787cd40" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36011 } }
I20260620 07:14:54.812117 28480 leader_election.cc:302] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b28075a4ad5460babfe98e71787cd40; no voters:
I20260620 07:14:54.812816 28480 leader_election.cc:288] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:54.812933 28490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:54.815445 28480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:54.816406 28490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [term 1 LEADER]: Becoming Leader. State: Replica: 2b28075a4ad5460babfe98e71787cd40, State: Running, Role: LEADER
I20260620 07:14:54.817104 28490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b28075a4ad5460babfe98e71787cd40" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36011 } }
I20260620 07:14:54.821074 28491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b28075a4ad5460babfe98e71787cd40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b28075a4ad5460babfe98e71787cd40" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36011 } } }
I20260620 07:14:54.821105 28492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b28075a4ad5460babfe98e71787cd40. Latest consensus state: current_term: 1 leader_uuid: "2b28075a4ad5460babfe98e71787cd40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b28075a4ad5460babfe98e71787cd40" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 36011 } } }
I20260620 07:14:54.821619 28491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:54.821650 28492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:54.822969 28500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:54.823489 28483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:54.807158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59959 (local address 127.6.53.126:36011)
0620 07:14:54.807355 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:54.807362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:54.807766 (+ 404us) server_negotiation.cc:407] Connection header received
0620 07:14:54.807955 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:54.807962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:54.808038 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:54.808148 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:54.809667 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:54.810450 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:54.811657 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:54.812151 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:54.812282 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:54.815144 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:54.815148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:54.815151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:54.815179 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:54.819508 (+ 4329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:54.820401 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:54.820406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:54.820410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:54.820591 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:54.821075 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:54.821080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:54.821086 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:54.821707 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:54.822075 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:54.822805 (+ 730us) server_negotiation.cc:299] Negotiation successful
0620 07:14:54.823127 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:14:54.829704 28500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:54.841382 28500 catalog_manager.cc:1347] Generated new cluster ID: 357ee390cdb74cdca4b52a3b9e788af9
I20260620 07:14:54.841987 28500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:54.854730 28500 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:54.856333 28500 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:54.875608 28500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b28075a4ad5460babfe98e71787cd40: Generated new TSK 0
I20260620 07:14:54.876482 28500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:54.890182 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 28413
I20260620 07:14:54.891698 28475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:14:54.983256 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 28413
2026-06-20T07:14:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2044 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:55 dist-test-slave-4n6h krb5kdc[28512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:55 dist-test-slave-4n6h krb5kdc[28512](info): set up 2 sockets
Jun 20 07:14:55 dist-test-slave-4n6h krb5kdc[28512](info): commencing operation
krb5kdc: starting...
W20260620 07:14:56.099512 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:56 dist-test-slave-4n6h krb5kdc[28512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:14:56.353955 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:35737
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:35737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:56.690933 28525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:56.691243 28525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:56.691460 28525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:56.699218 28525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:56.699350 28525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:56.699478 28525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:56.699570 28525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:56.699666 28525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:56.710192 28525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:35737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:35737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:56.711593 28525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:56.712945 28525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:56.725510 28531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:56.725657 28530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:56.725638 28533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:56.726624 28525 server_base.cc:1034] running on GCE node
I20260620 07:14:56.727360 28525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:56.729045 28525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:56.730301 28525 hybrid_clock.cc:648] HybridClock initialized: now 1781939696730198 us; error 65 us; skew 500 ppm
I20260620 07:14:56.730719 28525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:56 dist-test-slave-4n6h krb5kdc[28512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939696, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:56.733894 28525 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:14:56.735528 28525 webserver.cc:460] Webserver started at http://127.6.53.126:43409/ using document root <none> and password file <none>
I20260620 07:14:56.736205 28525 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:56.736352 28525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:56.736660 28525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:56.739732 28525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "83623813219b44448dc9eb948b19e639"
format_stamp: "Formatted at 2026-06-20 07:14:56 on dist-test-slave-4n6h"
server_key: "2f8289425610db9a00d39acb83710e35"
server_key_iv: "2b68128c8d154fc695fac091045c3730"
server_key_version: "encryptionkey@0"
I20260620 07:14:56.740480 28525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "83623813219b44448dc9eb948b19e639"
format_stamp: "Formatted at 2026-06-20 07:14:56 on dist-test-slave-4n6h"
server_key: "2f8289425610db9a00d39acb83710e35"
server_key_iv: "2b68128c8d154fc695fac091045c3730"
server_key_version: "encryptionkey@0"
I20260620 07:14:56.745671 28525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:14:56.749333 28540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:56.750854 28525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:14:56.751053 28525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "83623813219b44448dc9eb948b19e639"
format_stamp: "Formatted at 2026-06-20 07:14:56 on dist-test-slave-4n6h"
server_key: "2f8289425610db9a00d39acb83710e35"
server_key_iv: "2b68128c8d154fc695fac091045c3730"
server_key_version: "encryptionkey@0"
I20260620 07:14:56.751274 28525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:56.772200 28525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:56.776808 28525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:56.777115 28525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:56.796026 28525 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:35737
I20260620 07:14:56.796077 28591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:35737 every 8 connection(s)
I20260620 07:14:56.797470 28525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:56.799204 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 28525
I20260620 07:14:56.799408 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:56.799854 6357 external_mini_cluster.cc:1428] Setting key 05a8a3687c3af1b02af9b0e1a95b241f
I20260620 07:14:56.802584 28592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:56.812866 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:56.801559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46541 (local address 127.6.53.126:35737)
0620 07:14:56.802384 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:56.802396 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:56.802437 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:14:56.804104 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:56.804126 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:56.804494 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:56.804928 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:56.805529 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:56.806941 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:56.808263 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:56.808937 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:56.809151 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:56.810421 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:56.810429 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:56.810446 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:56.811125 (+ 679us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:56.811223 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:56.811259 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:56.811421 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:56.811643 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 07:14:56.811932 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":346,"threads_started":1}
I20260620 07:14:56.813823 28592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639: Bootstrap starting.
I20260620 07:14:56.816949 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:56.817335 28592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:56.818384 28592 log.cc:824] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:56.821399 28592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639: No bootstrap required, opened a new log
I20260620 07:14:56.828194 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:56.821759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38151 (local address 127.6.53.126:35737)
0620 07:14:56.821963 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:56.821971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:56.822439 (+ 468us) server_negotiation.cc:407] Connection header received
0620 07:14:56.822675 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:56.822680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:56.822740 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:56.822845 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:56.823368 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:56.824150 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:56.825495 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:56.825973 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:56.826139 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:56.827328 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:56.827333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:56.827336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:56.827357 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:56.827395 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:56.827410 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:56.827612 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:56.827754 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 07:14:56.827858 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:14:56.830099 28592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83623813219b44448dc9eb948b19e639" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35737 } }
I20260620 07:14:56.830561 28592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:56.830680 28592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83623813219b44448dc9eb948b19e639, State: Initialized, Role: FOLLOWER
I20260620 07:14:56.831326 28592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83623813219b44448dc9eb948b19e639" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35737 } }
I20260620 07:14:56.831573 28592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:56.831728 28592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:56.831916 28592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:56.834422 28592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83623813219b44448dc9eb948b19e639" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35737 } }
I20260620 07:14:56.834982 28592 leader_election.cc:302] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83623813219b44448dc9eb948b19e639; no voters:
I20260620 07:14:56.835558 28592 leader_election.cc:288] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:56.835717 28602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:56.836409 28602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [term 1 LEADER]: Becoming Leader. State: Replica: 83623813219b44448dc9eb948b19e639, State: Running, Role: LEADER
I20260620 07:14:56.836932 28602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83623813219b44448dc9eb948b19e639" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35737 } }
I20260620 07:14:56.837651 28592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:56.839947 28604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83623813219b44448dc9eb948b19e639. Latest consensus state: current_term: 1 leader_uuid: "83623813219b44448dc9eb948b19e639" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83623813219b44448dc9eb948b19e639" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35737 } } }
I20260620 07:14:56.839972 28603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83623813219b44448dc9eb948b19e639" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83623813219b44448dc9eb948b19e639" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35737 } } }
I20260620 07:14:56.840535 28604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:56.840559 28603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:56.843829 28607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:56.849120 28607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:56.857923 28607 catalog_manager.cc:1347] Generated new cluster ID: 6eb637d718d343158c767a4921ff8e11
I20260620 07:14:56.858065 28607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:56.867786 28607 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:56.869256 28607 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:56.880650 28607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83623813219b44448dc9eb948b19e639: Generated new TSK 0
I20260620 07:14:56.881649 28607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:56.887136 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 28525
W20260620 07:14:56.888598 28587 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46541
W20260620 07:14:56.889426 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 28525: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:14:56.889623 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 28525
2026-06-20T07:14:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:56 dist-test-slave-4n6h krb5kdc[28623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:56 dist-test-slave-4n6h krb5kdc[28623](info): set up 2 sockets
Jun 20 07:14:56 dist-test-slave-4n6h krb5kdc[28623](info): commencing operation
krb5kdc: starting...
W20260620 07:14:58.050705 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:14:58 dist-test-slave-4n6h krb5kdc[28623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:14:58.291482 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:40957
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:34047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:40957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:58.608515 28636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:58.608839 28636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:58.609002 28636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:58.616134 28636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:58.616272 28636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:58.616456 28636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:58.616585 28636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:58.616663 28636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:58.626148 28636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:34047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:40957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:40957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:14:58.627426 28636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:58.628655 28636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:58.640517 28644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:58.640524 28641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:58.641155 28642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:58.641664 28636 server_base.cc:1034] running on GCE node
I20260620 07:14:58.642642 28636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:58.644409 28636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:58.645637 28636 hybrid_clock.cc:648] HybridClock initialized: now 1781939698645592 us; error 30 us; skew 500 ppm
I20260620 07:14:58.646005 28636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:14:58 dist-test-slave-4n6h krb5kdc[28623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939698, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:58.653321 28636 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:14:58.654811 28636 webserver.cc:460] Webserver started at http://127.6.53.126:37291/ using document root <none> and password file <none>
I20260620 07:14:58.655494 28636 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:58.655642 28636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:58.655939 28636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:58.658911 28636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "de54f1496f944f8b91658ba43d2ed537"
format_stamp: "Formatted at 2026-06-20 07:14:58 on dist-test-slave-4n6h"
server_key: "036e669e3398ec08be5cc73e75196d3f"
server_key_iv: "086c152eac77a8848d6428d8d40fc826"
server_key_version: "encryptionkey@0"
I20260620 07:14:58.659667 28636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "de54f1496f944f8b91658ba43d2ed537"
format_stamp: "Formatted at 2026-06-20 07:14:58 on dist-test-slave-4n6h"
server_key: "036e669e3398ec08be5cc73e75196d3f"
server_key_iv: "086c152eac77a8848d6428d8d40fc826"
server_key_version: "encryptionkey@0"
I20260620 07:14:58.664810 28636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:14:58.668146 28651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:58.669324 28636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:14:58.669519 28636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "de54f1496f944f8b91658ba43d2ed537"
format_stamp: "Formatted at 2026-06-20 07:14:58 on dist-test-slave-4n6h"
server_key: "036e669e3398ec08be5cc73e75196d3f"
server_key_iv: "086c152eac77a8848d6428d8d40fc826"
server_key_version: "encryptionkey@0"
I20260620 07:14:58.669727 28636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:58.675338 28636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:58.679256 28636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:58.679538 28636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:58.698043 28636 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:40957
I20260620 07:14:58.698094 28702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:40957 every 8 connection(s)
I20260620 07:14:58.699534 28636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:58.703059 28703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:58.703830 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 28636
I20260620 07:14:58.704013 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:14:58.704485 6357 external_mini_cluster.cc:1428] Setting key 29444cb419b2c6229476ed145f334715
I20260620 07:14:58.713202 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:58.706274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44799 (local address 127.6.53.126:40957)
0620 07:14:58.707105 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:58.707120 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:58.707165 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:14:58.709584 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:58.709610 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:58.710050 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:58.710530 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:58.711450 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:58.711458 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:58.711474 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:58.711489 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:58.711549 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:58.711576 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:58.711769 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:58.711890 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 07:14:58.712071 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":335,"threads_started":1}
I20260620 07:14:58.716763 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:58.717258 28703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537: Bootstrap starting.
I20260620 07:14:58.720217 28703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:58.721343 28703 log.cc:824] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:58.723366 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:58.720989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35333 (local address 127.6.53.126:40957)
0620 07:14:58.721166 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:58.721181 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:58.721536 (+ 355us) server_negotiation.cc:407] Connection header received
0620 07:14:58.721700 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:58.721706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:58.721759 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:58.721859 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:58.722767 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:58.722772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:58.722789 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:14:58.722805 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:58.722847 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:14:58.722861 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:58.722974 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:58.723068 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 07:14:58.723119 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:14:58.724071 28703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537: No bootstrap required, opened a new log
I20260620 07:14:58.731812 28703 raft_consensus.cc:348] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de54f1496f944f8b91658ba43d2ed537" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40957 } }
I20260620 07:14:58.732173 28703 raft_consensus.cc:374] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:58.732277 28703 raft_consensus.cc:729] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de54f1496f944f8b91658ba43d2ed537, State: Initialized, Role: FOLLOWER
I20260620 07:14:58.732780 28703 consensus_queue.cc:260] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de54f1496f944f8b91658ba43d2ed537" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40957 } }
I20260620 07:14:58.733014 28703 raft_consensus.cc:388] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:58.733158 28703 raft_consensus.cc:482] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:58.733333 28703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:58.735735 28703 raft_consensus.cc:504] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de54f1496f944f8b91658ba43d2ed537" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40957 } }
I20260620 07:14:58.736207 28703 leader_election.cc:302] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de54f1496f944f8b91658ba43d2ed537; no voters:
I20260620 07:14:58.736737 28703 leader_election.cc:288] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:58.736836 28713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:58.737500 28713 raft_consensus.cc:686] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [term 1 LEADER]: Becoming Leader. State: Replica: de54f1496f944f8b91658ba43d2ed537, State: Running, Role: LEADER
I20260620 07:14:58.737946 28713 consensus_queue.cc:237] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de54f1496f944f8b91658ba43d2ed537" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40957 } }
I20260620 07:14:58.738685 28703 sys_catalog.cc:564] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:58.740777 28715 sys_catalog.cc:455] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de54f1496f944f8b91658ba43d2ed537. Latest consensus state: current_term: 1 leader_uuid: "de54f1496f944f8b91658ba43d2ed537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de54f1496f944f8b91658ba43d2ed537" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40957 } } }
I20260620 07:14:58.741220 28715 sys_catalog.cc:458] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:58.741576 28714 sys_catalog.cc:455] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de54f1496f944f8b91658ba43d2ed537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de54f1496f944f8b91658ba43d2ed537" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 40957 } } }
I20260620 07:14:58.742340 28714 sys_catalog.cc:458] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:58.743300 28722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:58.748683 28722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:58.757161 28722 catalog_manager.cc:1347] Generated new cluster ID: b22e70c7fc9e47cbaffc81f78a401966
I20260620 07:14:58.757258 28722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:58.773413 28722 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:58.773953 28722 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:58.779007 28722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de54f1496f944f8b91658ba43d2ed537: Generated new TSK 0
I20260620 07:14:58.779711 28722 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:58.789778 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 28636
W20260620 07:14:58.790812 28698 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44799
W20260620 07:14:58.791523 6357 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 28636: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:14:58.791677 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 28636
2026-06-20T07:14:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1900 ms)
[----------] 18 tests from AuthTokenIssuingTest (34514 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:58 dist-test-slave-4n6h krb5kdc[28734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:58 dist-test-slave-4n6h krb5kdc[28734](info): set up 2 sockets
Jun 20 07:14:58 dist-test-slave-4n6h krb5kdc[28734](info): commencing operation
krb5kdc: starting...
W20260620 07:14:59.882882 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:15:00 dist-test-slave-4n6h krb5kdc[28734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:15:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:15:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:15:00.135814 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:44433
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:35063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:44433,127.6.53.125:45329,127.6.53.124:37695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:00.464958 28747 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:15:00.465337 28747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:00.465477 28747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:00.465643 28747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:00.473054 28747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:00.473184 28747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:00.473316 28747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:00.473416 28747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:00.473526 28747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:00.483266 28747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:35063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:44433,127.6.53.125:45329,127.6.53.124:37695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:44433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:15:00.484546 28747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:00.485859 28747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:00.497648 28752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:00.497670 28755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:00.497694 28753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:00.498509 28747 server_base.cc:1034] running on GCE node
I20260620 07:15:00.499172 28747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:00.500437 28747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:00.501658 28747 hybrid_clock.cc:648] HybridClock initialized: now 1781939700501592 us; error 39 us; skew 500 ppm
I20260620 07:15:00.501946 28747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:15:00 dist-test-slave-4n6h krb5kdc[28734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:00.505046 28747 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:15:00.506542 28747 webserver.cc:460] Webserver started at http://127.6.53.126:42885/ using document root <none> and password file <none>
I20260620 07:15:00.507205 28747 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:00.507292 28747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:00.507555 28747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:00.510630 28747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "464193a40c18499e9320463db53a574e"
format_stamp: "Formatted at 2026-06-20 07:15:00 on dist-test-slave-4n6h"
server_key: "741a9e90c375fd50b698c3d01e79df6d"
server_key_iv: "cd4d5964af69a54a2341597cab21517f"
server_key_version: "encryptionkey@0"
I20260620 07:15:00.511377 28747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "464193a40c18499e9320463db53a574e"
format_stamp: "Formatted at 2026-06-20 07:15:00 on dist-test-slave-4n6h"
server_key: "741a9e90c375fd50b698c3d01e79df6d"
server_key_iv: "cd4d5964af69a54a2341597cab21517f"
server_key_version: "encryptionkey@0"
I20260620 07:15:00.516726 28747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 07:15:00.520511 28762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:00.521942 28747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:15:00.522084 28747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "464193a40c18499e9320463db53a574e"
format_stamp: "Formatted at 2026-06-20 07:15:00 on dist-test-slave-4n6h"
server_key: "741a9e90c375fd50b698c3d01e79df6d"
server_key_iv: "cd4d5964af69a54a2341597cab21517f"
server_key_version: "encryptionkey@0"
I20260620 07:15:00.522358 28747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:00.542372 28747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:00.546768 28747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:00.546991 28747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:00.565188 28747 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:44433
I20260620 07:15:00.565228 28813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:44433 every 8 connection(s)
I20260620 07:15:00.566583 28747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:00.569802 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 28747
I20260620 07:15:00.570029 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:15:00.570596 28814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:00.570820 6357 external_mini_cluster.cc:1428] Setting key 5e30b4bae95fd77a9cb2e9fa3453f547
I20260620 07:15:00.577382 28814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:15:00 dist-test-slave-4n6h krb5kdc[28734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.53.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.125@KRBTEST.COM" created.
I20260620 07:15:00.599630 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.580624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:44433 (local address 127.0.0.1:41252)
0620 07:15:00.581814 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:00.581846 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:00.583576 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:00.584383 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:00.584389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:00.585075 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:00.585408 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:00.585424 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.587061 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.587068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:00.588112 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:00.588123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.589016 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.589022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:00.589202 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:00.589857 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:00.589879 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:00.591998 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:00.595372 (+ 3374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:00.595385 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:00.595398 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:00.595761 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:00.596165 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:00.596181 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:00.596185 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:00.596340 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:00.597147 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:00.597161 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:00.597373 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 07:15:00.597634 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 07:15:00.598303 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":158,"thread_start_us":782,"threads_started":1}
I20260620 07:15:00.599607 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.580623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.6.53.126:44433)
0620 07:15:00.581498 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:00.581514 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:00.581972 (+ 458us) server_negotiation.cc:407] Connection header received
0620 07:15:00.583919 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:00.583924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:00.584165 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:00.584370 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:00.585623 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.586900 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.588257 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.588791 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.588993 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:00.592212 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:00.592220 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:00.592231 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:00.592261 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:00.595251 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:00.595893 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:00.595903 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:00.595911 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:00.596053 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:00.596445 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:00.596451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:00.596454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:00.597001 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:00.597207 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:00.597754 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 07:15:00.598167 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":601,"thread_start_us":397,"threads_started":1}
I20260620 07:15:00.602584 28814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:15:00.603817 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.603299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:45329 (local address 127.0.0.1:42492)
0620 07:15:00.603480 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:00.603642 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 07:15:00.604146 28764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.53.125:45329: connect: Connection refused (error 111)
I20260620 07:15:00.605815 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.605335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:45329 (local address 127.0.0.1:42504)
0620 07:15:00.605529 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:00.605630 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 07:15:00.606065 28814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.125:45329: Network error: Client connection negotiation failed: client connection to 127.6.53.125:45329: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.125.keytab.
Entry for principal kudu/127.6.53.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.125.keytab.
I20260620 07:15:00.625422 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.125:45329
--webserver_interface=127.6.53.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.125.keytab
--principal=kudu/127.6.53.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:35063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:44433,127.6.53.125:45329,127.6.53.124:37695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:15:00.655428 28814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } attempt: 1
I20260620 07:15:00.656699 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.656115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:45329 (local address 127.0.0.1:42520)
0620 07:15:00.656360 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:00.656504 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 07:15:00.657943 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.657477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:45329 (local address 127.0.0.1:42524)
0620 07:15:00.657644 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:00.657747 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 07:15:00.658149 28814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.125:45329: Network error: Client connection negotiation failed: client connection to 127.6.53.125:45329: connect: Connection refused (error 111)
I20260620 07:15:00.726439 28814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } attempt: 2
I20260620 07:15:00.727819 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.727168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:45329 (local address 127.0.0.1:42536)
0620 07:15:00.727507 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:00.727633 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 07:15:00.729023 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.728552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:45329 (local address 127.0.0.1:42548)
0620 07:15:00.728713 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:00.728826 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 07:15:00.729279 28814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.125:45329: Network error: Client connection negotiation failed: client connection to 127.6.53.125:45329: connect: Connection refused (error 111)
I20260620 07:15:00.820575 28814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } attempt: 3
I20260620 07:15:00.821933 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.821277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:45329 (local address 127.0.0.1:42550)
0620 07:15:00.821550 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:00.821692 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 07:15:00.823292 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.822762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:45329 (local address 127.0.0.1:42562)
0620 07:15:00.822965 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:00.823082 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 07:15:00.823537 28814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.125:45329: Network error: Client connection negotiation failed: client connection to 127.6.53.125:45329: connect: Connection refused (error 111)
I20260620 07:15:00.966828 28814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } attempt: 4
I20260620 07:15:00.968176 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.967537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:45329 (local address 127.0.0.1:42570)
0620 07:15:00.967882 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:00.968002 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260620 07:15:00.969231 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.968814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:45329 (local address 127.0.0.1:42580)
0620 07:15:00.968971 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:00.969074 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:45329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 07:15:00.969422 28814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.125:45329: Network error: Client connection negotiation failed: client connection to 127.6.53.125:45329: connect: Connection refused (error 111)
W20260620 07:15:00.971458 28821 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:15:00.971761 28821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:00.971843 28821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:00.971935 28821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:00.979557 28821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:00.979648 28821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:00.979713 28821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:00.979774 28821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:00.979837 28821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:00.989856 28821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:35063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:44433,127.6.53.125:45329,127.6.53.124:37695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.125.keytab
--principal=kudu/127.6.53.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.125:45329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.53.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:15:00.991161 28821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:00.992535 28821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:01.004596 28826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:01.004846 28827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:01.005283 28829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:01.006296 28821 server_base.cc:1034] running on GCE node
I20260620 07:15:01.007131 28821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:01.008574 28821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:01.009824 28821 hybrid_clock.cc:648] HybridClock initialized: now 1781939701009752 us; error 47 us; skew 500 ppm
I20260620 07:15:01.010149 28821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:15:01 dist-test-slave-4n6h krb5kdc[28734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939701, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:01.013353 28821 init.cc:377] Logged in from keytab as kudu/127.6.53.125@KRBTEST.COM (short username kudu)
I20260620 07:15:01.014882 28821 webserver.cc:460] Webserver started at http://127.6.53.125:37621/ using document root <none> and password file <none>
I20260620 07:15:01.015563 28821 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:01.015693 28821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:01.016001 28821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:01.019402 28821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/data/instance:
uuid: "92b47d1691e44e648ed0bcd88acec82c"
format_stamp: "Formatted at 2026-06-20 07:15:01 on dist-test-slave-4n6h"
server_key: "3335e05e6663bdba60a21b901df4f1ac"
server_key_iv: "762f7e77ca5da2248f2d96e69cc771c7"
server_key_version: "encryptionkey@0"
I20260620 07:15:01.020201 28821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/wal/instance:
uuid: "92b47d1691e44e648ed0bcd88acec82c"
format_stamp: "Formatted at 2026-06-20 07:15:01 on dist-test-slave-4n6h"
server_key: "3335e05e6663bdba60a21b901df4f1ac"
server_key_iv: "762f7e77ca5da2248f2d96e69cc771c7"
server_key_version: "encryptionkey@0"
I20260620 07:15:01.025223 28821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 07:15:01.028846 28836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:01.030187 28821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:15:01.030411 28821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/wal
uuid: "92b47d1691e44e648ed0bcd88acec82c"
format_stamp: "Formatted at 2026-06-20 07:15:01 on dist-test-slave-4n6h"
server_key: "3335e05e6663bdba60a21b901df4f1ac"
server_key_iv: "762f7e77ca5da2248f2d96e69cc771c7"
server_key_version: "encryptionkey@0"
I20260620 07:15:01.030642 28821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:01.045012 28821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:01.049525 28821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:01.049820 28821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:01.068560 28821 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.125:45329
I20260620 07:15:01.068583 28887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.125:45329 every 8 connection(s)
I20260620 07:15:01.069912 28821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/data/info.pb
I20260620 07:15:01.073117 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 28821
I20260620 07:15:01.073352 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-1/wal/instance
I20260620 07:15:01.073621 28888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:01.073824 6357 external_mini_cluster.cc:1428] Setting key 191fca744c4997904a8831ba37dedb86
I20260620 07:15:01.080085 28888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:15:01 dist-test-slave-4n6h krb5kdc[28734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939701, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.125@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.53.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.124@KRBTEST.COM" created.
I20260620 07:15:01.102355 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.082899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.6.53.126:44433)
0620 07:15:01.083136 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.083144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.084305 (+ 1161us) server_negotiation.cc:407] Connection header received
0620 07:15:01.086194 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.086204 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.086320 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.086439 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.087885 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.089076 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.090679 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.091473 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.091628 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.095357 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.095364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.095369 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.095415 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.098819 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.099661 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.099667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.099670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.099819 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.100282 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.100290 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.100295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.100715 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.100879 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.101586 (+ 707us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.101865 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:15:01.103155 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.083315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:44433 (local address 127.0.0.1:41264)
0620 07:15:01.084108 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.084158 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:01.085924 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:01.086666 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:01.086674 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:01.087155 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:01.087702 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.087718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.089239 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.089242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.090526 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.090538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.091681 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.091685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.091885 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.092815 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:01.092839 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:01.095093 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:01.098989 (+ 3896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.098996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.099011 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.099529 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.099947 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.099951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.099953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.100165 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.100849 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:01.100855 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:01.101214 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 07:15:01.101445 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 07:15:01.101837 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":373,"threads_started":1}
I20260620 07:15:01.105095 28888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:15:01 dist-test-slave-4n6h krb5kdc[28734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939701, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.125@KRBTEST.COM for kudu/127.6.53.125@KRBTEST.COM
I20260620 07:15:01.121739 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.105752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:45329 (local address 127.0.0.1:42594)
0620 07:15:01.105950 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.105972 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:01.106073 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:01.107194 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:01.107201 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:01.107651 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:01.107793 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.107802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.109914 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.109919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.110860 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.110871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.112089 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.112093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.112226 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.112863 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:01.112882 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:01.114908 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:01.119333 (+ 4425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.119336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.119339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.119699 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.120123 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.120126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.120128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.120254 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.121027 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:01.121030 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:01.121166 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 07:15:01.121276 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 07:15:01.121481 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 07:15:01.122314 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.106056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42594 (local address 127.6.53.125:45329)
0620 07:15:01.106767 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.106772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.106798 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:15:01.106840 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.106846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.107021 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.107139 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.107979 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.109761 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.111001 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.111951 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.112097 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.115148 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.115165 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.115185 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.115226 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.119204 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.119847 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.119859 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.119865 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.120014 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.120358 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.120366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.120372 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.120871 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.121109 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.121486 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.121791 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":386,"threads_started":1}
I20260620 07:15:01.124881 28888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:15:01.126076 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.125599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:55360)
0620 07:15:01.125791 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.125903 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 07:15:01.126407 28838 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
I20260620 07:15:01.127935 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.127605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:55374)
0620 07:15:01.127732 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.127810 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 07:15:01.128206 28888 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:37695: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.124.keytab.
Entry for principal kudu/127.6.53.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.124.keytab.
I20260620 07:15:01.133086 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.124:37695
--webserver_interface=127.6.53.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.124.keytab
--principal=kudu/127.6.53.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:35063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:44433,127.6.53.125:45329,127.6.53.124:37695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:15:01.177551 28888 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } attempt: 1
I20260620 07:15:01.178797 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.178370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:55388)
0620 07:15:01.178560 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.178646 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 07:15:01.179792 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.179463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:55390)
0620 07:15:01.179615 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.179695 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 07:15:01.179973 28888 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:37695: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
I20260620 07:15:01.248176 28888 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } attempt: 2
I20260620 07:15:01.249356 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.248818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:55396)
0620 07:15:01.249053 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.249166 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 07:15:01.250464 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.250056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:55398)
0620 07:15:01.250210 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.250317 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 07:15:01.250706 28888 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:37695: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
I20260620 07:15:01.268666 28814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } attempt: 5
Jun 20 07:15:01 dist-test-slave-4n6h krb5kdc[28734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for kudu/127.6.53.125@KRBTEST.COM
I20260620 07:15:01.282171 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.269403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:45329 (local address 127.0.0.1:42596)
0620 07:15:01.269618 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.269640 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:01.269791 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:01.270145 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:01.270151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:01.270588 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:01.270759 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.270768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.271761 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.271768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.272710 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.272722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.273411 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.273416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.273611 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.274289 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:01.274318 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:01.276598 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:01.279735 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.279740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.279745 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.280113 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.280502 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.280507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.280510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.280652 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.281168 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:01.281174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:01.281336 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 07:15:01.281480 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 07:15:01.281788 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 07:15:01.282433 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.269636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42596 (local address 127.6.53.125:45329)
0620 07:15:01.269828 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.269836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.269859 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:15:01.269921 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.269932 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.270005 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.270110 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.270910 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.271630 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.272848 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.273269 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.273400 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.276804 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.276809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.276812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.276844 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.279571 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.280235 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.280243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.280249 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.280381 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.280762 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.280768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.280771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.281053 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.281202 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.281567 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.281793 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:15:01.283411 28814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:15:01.284428 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.284033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:55412)
0620 07:15:01.284211 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.284296 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 07:15:01.285279 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.284978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:55426)
0620 07:15:01.285108 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.285191 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 07:15:01.285501 28814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:37695: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
I20260620 07:15:01.336714 28814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } attempt: 1
I20260620 07:15:01.338016 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.337417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:55428)
0620 07:15:01.337691 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.337805 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 07:15:01.339113 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.338697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:55436)
0620 07:15:01.338863 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.338960 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 07:15:01.339404 28814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:37695: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
I20260620 07:15:01.341934 28888 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } attempt: 3
I20260620 07:15:01.343080 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.342509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:55448)
0620 07:15:01.342748 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.342867 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 07:15:01.344381 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.343832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:55450)
0620 07:15:01.344092 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.344204 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260620 07:15:01.344662 28888 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:37695: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
I20260620 07:15:01.407641 28814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } attempt: 2
I20260620 07:15:01.408952 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.408436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:55460)
0620 07:15:01.408689 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.408789 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 07:15:01.410087 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.409750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:55474)
0620 07:15:01.409910 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.409983 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 07:15:01.410414 28814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:37695: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
W20260620 07:15:01.469058 28895 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:15:01.469457 28895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:01.469588 28895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:01.469729 28895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:01.477407 28895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:01.477537 28895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:01.477638 28895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:01.477735 28895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:01.477828 28895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:01.487910 28888 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } attempt: 4
I20260620 07:15:01.488148 28895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:35063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:44433,127.6.53.125:45329,127.6.53.124:37695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.124.keytab
--principal=kudu/127.6.53.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.124:37695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.53.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:15:01.489207 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.488662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:57274)
0620 07:15:01.488924 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.489027 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 07:15:01.489849 28895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:01.490289 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.489869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:57278)
0620 07:15:01.490049 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.490127 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 07:15:01.490566 28888 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:37695: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
I20260620 07:15:01.491372 28895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:01.506815 28903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:01.506839 28900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:01.506861 28901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:01.508512 28895 server_base.cc:1034] running on GCE node
I20260620 07:15:01.509312 28895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:01.510696 28895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:01.511900 28895 hybrid_clock.cc:648] HybridClock initialized: now 1781939701511813 us; error 63 us; skew 500 ppm
I20260620 07:15:01.512231 28895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:15:01 dist-test-slave-4n6h krb5kdc[28734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939701, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:01.515525 28895 init.cc:377] Logged in from keytab as kudu/127.6.53.124@KRBTEST.COM (short username kudu)
I20260620 07:15:01.516602 28814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } attempt: 3
I20260620 07:15:01.517093 28895 webserver.cc:460] Webserver started at http://127.6.53.124:45235/ using document root <none> and password file <none>
I20260620 07:15:01.517621 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.517198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:57292)
0620 07:15:01.517381 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.517467 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 07:15:01.517912 28895 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:01.518074 28895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:01.518501 28895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:01.518749 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.518389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:57298)
0620 07:15:01.518544 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.518652 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 07:15:01.519017 28814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:37695: Network error: Client connection negotiation failed: client connection to 127.6.53.124:37695: connect: Connection refused (error 111)
I20260620 07:15:01.521566 28895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/data/instance:
uuid: "3148417446a84e4b9e8fffb00c9067a3"
format_stamp: "Formatted at 2026-06-20 07:15:01 on dist-test-slave-4n6h"
server_key: "33f6a25fa5a04f91690569ea579e7c1d"
server_key_iv: "ee16a94ad775c22e08d9916650ad3a52"
server_key_version: "encryptionkey@0"
I20260620 07:15:01.522428 28895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/wal/instance:
uuid: "3148417446a84e4b9e8fffb00c9067a3"
format_stamp: "Formatted at 2026-06-20 07:15:01 on dist-test-slave-4n6h"
server_key: "33f6a25fa5a04f91690569ea579e7c1d"
server_key_iv: "ee16a94ad775c22e08d9916650ad3a52"
server_key_version: "encryptionkey@0"
I20260620 07:15:01.527678 28895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:15:01.531148 28910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:01.532461 28895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 07:15:01.532737 28895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/wal
uuid: "3148417446a84e4b9e8fffb00c9067a3"
format_stamp: "Formatted at 2026-06-20 07:15:01 on dist-test-slave-4n6h"
server_key: "33f6a25fa5a04f91690569ea579e7c1d"
server_key_iv: "ee16a94ad775c22e08d9916650ad3a52"
server_key_version: "encryptionkey@0"
I20260620 07:15:01.532964 28895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:01.584614 28895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:01.589010 28895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:01.589286 28895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:01.608091 28895 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.124:37695
I20260620 07:15:01.608129 28961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.124:37695 every 8 connection(s)
I20260620 07:15:01.609357 28895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/data/info.pb
I20260620 07:15:01.612798 28962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:01.618360 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 28895
I20260620 07:15:01.618548 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939352012798-6357-0/minicluster-data/master-2/wal/instance
I20260620 07:15:01.618985 6357 external_mini_cluster.cc:1428] Setting key 19dc88758f8a65bb432f43c07db45637
I20260620 07:15:01.619264 28962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:15:01 dist-test-slave-4n6h krb5kdc[28734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939701, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.124@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
Jun 20 07:15:01 dist-test-slave-4n6h krb5kdc[28734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.124@KRBTEST.COM
I20260620 07:15:01.638883 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.622014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.6.53.126:44433)
0620 07:15:01.622569 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.622575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.623075 (+ 500us) server_negotiation.cc:407] Connection header received
0620 07:15:01.624057 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.624061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.624122 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.624202 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.626015 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.627023 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.628479 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.628860 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.628955 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.632566 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.632571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.632574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.632605 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.635528 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.636349 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.636355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.636362 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.636508 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.636963 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.636967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.636971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.637346 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.637506 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.638234 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.638478 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":256,"threads_started":1}
I20260620 07:15:01.639817 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.622078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:44433 (local address 127.0.0.1:37312)
0620 07:15:01.622914 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.622952 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:01.623801 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:01.624291 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:01.624298 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:01.625058 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:01.625602 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.625644 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.627185 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.627192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.628337 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.628354 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.629069 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.629076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.629243 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.629922 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:01.629949 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:01.632369 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:01.635659 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.635670 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.635680 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.636234 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.636631 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.636635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.636639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.636842 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.637574 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:01.637580 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:01.637899 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 07:15:01.638159 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 07:15:01.638623 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":461,"threads_started":1}
I20260620 07:15:01.639845 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.620756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57300 (local address 127.6.53.124:37695)
0620 07:15:01.621493 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.621506 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.621554 (+ 48us) server_negotiation.cc:407] Connection header received
0620 07:15:01.623801 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.623825 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.624111 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.624306 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.625747 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.627144 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.628384 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.628900 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.629098 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.633427 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.633434 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.633444 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.633475 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.636473 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.637328 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.637338 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.637345 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.637461 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.637844 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.637849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.637852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.638169 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.638499 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.638842 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.639097 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":322,"threads_started":1}
I20260620 07:15:01.641348 28962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:15:01 dist-test-slave-4n6h krb5kdc[28734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939701, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.124@KRBTEST.COM for kudu/127.6.53.125@KRBTEST.COM
Jun 20 07:15:01 dist-test-slave-4n6h krb5kdc[28734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:15:01.654505 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.641988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:45329 (local address 127.0.0.1:60576)
0620 07:15:01.642206 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.642224 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:01.642341 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:01.642835 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:01.642840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:01.643162 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:01.643322 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.643333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.644240 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.644247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.645242 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.645256 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.645928 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.645934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.646151 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.646705 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:01.646721 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:01.648833 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:01.652154 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.652159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.652162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.652513 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.652875 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.652881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.652885 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.653033 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.653547 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:01.653552 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:01.653699 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 07:15:01.653845 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 07:15:01.654123 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260620 07:15:01.654565 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.642093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.6.53.125:45329)
0620 07:15:01.642236 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.642240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.642296 (+ 56us) server_negotiation.cc:407] Connection header received
0620 07:15:01.642476 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.642484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.642571 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.642667 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.643442 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.644105 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.645368 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.645786 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.645876 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.649028 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.649033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.649036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.649066 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.652025 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.652630 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.652636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.652640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.652767 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.653153 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.653158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.653161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.653396 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.653582 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.653934 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.654150 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 07:15:01.655475 28962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:15:01.656903 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.642733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.6.53.126:44433)
0620 07:15:01.643089 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.643093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.643105 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:15:01.643139 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.643142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.643201 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.643270 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.644457 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.645090 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.646576 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.647006 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.647103 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.651494 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.651498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.651501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.651529 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.654037 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.654773 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.654778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.654780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.654923 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.655489 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.655494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.655498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.655811 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.656099 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.656383 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.656591 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270}
Jun 20 07:15:01 dist-test-slave-4n6h krb5kdc[28734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939701, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.124@KRBTEST.COM for kudu/127.6.53.124@KRBTEST.COM
Jun 20 07:15:01 dist-test-slave-4n6h krb5kdc[28734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.125@KRBTEST.COM
I20260620 07:15:01.668341 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.656062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:57306)
0620 07:15:01.656200 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.656215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:01.656296 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:01.656599 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:01.656603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:01.656937 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:01.657084 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.657093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.658142 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.658150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.659611 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.659629 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.660412 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.660416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.660591 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.661128 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:01.661148 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:01.663406 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:01.666134 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.666142 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.666146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.666560 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.666920 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.666925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.666929 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.667049 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.667557 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:01.667561 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:01.667671 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 07:15:01.667770 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 07:15:01.667966 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 07:15:01.668542 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.656259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57306 (local address 127.6.53.124:37695)
0620 07:15:01.656392 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.656397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.656410 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:15:01.656443 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.656449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.656496 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.656588 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.657216 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.658029 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.659785 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.660225 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.660468 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.663583 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.663587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.663590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.663614 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.665992 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.666667 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.666675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.666679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.666808 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.667154 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.667159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.667162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.667407 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.667560 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.667921 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.668211 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:15:01.673084 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.658562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.6.53.125:45329)
0620 07:15:01.658717 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.658720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.658736 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:15:01.658771 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.658774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.658819 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.658891 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.660357 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.661123 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.662798 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.663250 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.663381 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.667560 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.667566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.667572 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.667608 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.670636 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.671335 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.671338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.671340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.671504 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.671903 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.671906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.671908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.672123 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.672294 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.672631 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.672803 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:15:01.673887 28962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3: Bootstrap starting.
I20260620 07:15:01.677331 28962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:01.677608 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:15:01.678483 28962 log.cc:824] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:01.681334 28962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3: No bootstrap required, opened a new log
I20260620 07:15:01.680537 28789 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "464193a40c18499e9320463db53a574e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37316
I20260620 07:15:01.692219 28962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } }
I20260620 07:15:01.692665 28962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3148417446a84e4b9e8fffb00c9067a3, State: Initialized, Role: FOLLOWER
I20260620 07:15:01.693193 28962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } }
I20260620 07:15:01.694099 28969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } } }
I20260620 07:15:01.694543 28969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:15:01.695215 28962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:01.696210 28814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } attempt: 4
Jun 20 07:15:01 dist-test-slave-4n6h krb5kdc[28734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for kudu/127.6.53.124@KRBTEST.COM
I20260620 07:15:01.712636 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.696957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:57320)
0620 07:15:01.697230 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.697255 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:01.697433 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:01.698050 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:01.698057 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:01.698621 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:01.698836 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.698851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.700339 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.700346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.701249 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.701260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.702264 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.702270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.702480 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.703133 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:01.703150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:01.705621 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:01.709623 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.709628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.709631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.710097 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.710546 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.710551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.710554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.710738 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.711620 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:01.711625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:01.711787 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 07:15:01.711948 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 07:15:01.712240 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260620 07:15:01.713585 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.697004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57320 (local address 127.6.53.124:37695)
0620 07:15:01.697197 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.697203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.697318 (+ 115us) server_negotiation.cc:407] Connection header received
0620 07:15:01.697628 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.697635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.697720 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.697833 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.699004 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.700174 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.701389 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.702106 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.702223 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.705791 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.705797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.705802 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.705842 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.709464 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.710223 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.710228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.710231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.710383 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.710880 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.710888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.710895 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.711364 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.711547 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.712929 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.713172 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 07:15:01.713730 28980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:15:01.713878 28980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:15:01.719305 28814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e: Bootstrap starting.
I20260620 07:15:01.710503 28895 master_runner.cc:186] Error getting master registration for 127.6.53.126:44433: OK, instance_id { permanent_uuid: "464193a40c18499e9320463db53a574e" instance_seqno: 1781939700559893 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:15:01.722473 28895 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:15:01.723277 28814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:01.724313 28814 log.cc:824] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:01.727384 28814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e: No bootstrap required, opened a new log
I20260620 07:15:01.732429 28814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } }
I20260620 07:15:01.732757 28814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 464193a40c18499e9320463db53a574e, State: Initialized, Role: FOLLOWER
I20260620 07:15:01.733191 28814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } }
I20260620 07:15:01.734405 28983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } } }
I20260620 07:15:01.735045 28983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:15:01.735261 28814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:01.735379 28789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 07:15:01.735562 28789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:01.739547 28789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } }
I20260620 07:15:01.741425 28789 leader_election.cc:288] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [CANDIDATE]: Term 1 election: Requested vote from peers 92b47d1691e44e648ed0bcd88acec82c (127.6.53.125:45329), 3148417446a84e4b9e8fffb00c9067a3 (127.6.53.124:37695)
I20260620 07:15:01.742636 28937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "464193a40c18499e9320463db53a574e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3148417446a84e4b9e8fffb00c9067a3"
I20260620 07:15:01.743280 28937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:01.748025 28937 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 464193a40c18499e9320463db53a574e in term 1.
I20260620 07:15:01.748796 28764 leader_election.cc:302] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3148417446a84e4b9e8fffb00c9067a3, 464193a40c18499e9320463db53a574e; no voters:
I20260620 07:15:01.750164 28983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:01.751515 28983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [term 1 LEADER]: Becoming Leader. State: Replica: 464193a40c18499e9320463db53a574e, State: Running, Role: LEADER
I20260620 07:15:01.752159 28983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } }
I20260620 07:15:01.761895 28991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 464193a40c18499e9320463db53a574e. Latest consensus state: current_term: 1 leader_uuid: "464193a40c18499e9320463db53a574e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } } }
I20260620 07:15:01.762498 28991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:01.775805 28995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:01.789765 28888 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } attempt: 5
W20260620 07:15:01.799897 28768 tablet.cc:2369] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:15:01.811961 28995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 07:15:01 dist-test-slave-4n6h krb5kdc[28734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939701, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.125@KRBTEST.COM for kudu/127.6.53.124@KRBTEST.COM
I20260620 07:15:01.824708 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.770112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.6.53.126:44433)
0620 07:15:01.770319 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.770326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.782889 (+ 12563us) server_negotiation.cc:407] Connection header received
0620 07:15:01.783216 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.783223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.783313 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.783397 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.794418 (+ 11021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.796073 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.802390 (+ 6317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.803724 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.803904 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.811535 (+ 7631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.811546 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.811554 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.811635 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.817007 (+ 5372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.818840 (+ 1833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.818850 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.818856 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.819216 (+ 360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.820236 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.820246 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.820252 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.821138 (+ 886us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.821757 (+ 619us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.822955 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.823669 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:15:01.836745 28937 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3 [term 1 FOLLOWER]: Refusing update from remote peer 464193a40c18499e9320463db53a574e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 07:15:01.841385 28916 tablet.cc:2369] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:15:01.846135 28991 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:01.851389 28747 master_runner.cc:186] Error getting master registration for 127.6.53.125:45329: OK, instance_id { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" instance_seqno: 1781939701063316 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:15:01.871430 28747 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:15:01.872038 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.790762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:37695 (local address 127.0.0.1:57342)
0620 07:15:01.791012 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.791031 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:01.791131 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:01.807482 (+ 16351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:01.807488 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:01.808032 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:01.808262 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.808276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.816672 (+ 8396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.816678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.818240 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.818293 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.819716 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.819723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.819982 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.820686 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:01.820716 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:01.823756 (+ 3040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:01.866955 (+ 43199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.866962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.866966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.867531 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.868069 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.868075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.868079 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.868248 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.870950 (+ 2702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:01.870957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:01.871124 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 07:15:01.871270 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 07:15:01.871607 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260620 07:15:01.873461 29004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.790481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.6.53.124:37695)
0620 07:15:01.804992 (+ 14511us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.804999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.805027 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:15:01.805090 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.805097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.805185 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.808446 (+ 3261us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.808715 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.810060 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.818505 (+ 8445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.819311 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.823943 (+ 4632us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.824217 (+ 274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.824224 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.824229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.824264 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.856292 (+ 32028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.867716 (+ 11424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.867722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.867726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.867920 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.868742 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.868748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.868751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.869188 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.872530 (+ 3342us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.872702 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.872952 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14346,"thread_start_us":14810,"threads_started":1}
I20260620 07:15:01.874752 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.783100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.6.53.125:45329)
0620 07:15:01.783291 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.783297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.787121 (+ 3824us) server_negotiation.cc:407] Connection header received
0620 07:15:01.788895 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.788902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.789001 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.789079 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.813337 (+ 24258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.814550 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.827475 (+ 12925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.828227 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.828330 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.861769 (+ 33439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.861777 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.861783 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.861833 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.865883 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.868420 (+ 2537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.868425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.868429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.868578 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.869692 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.869698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.869702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.870065 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.870384 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.874085 (+ 3701us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.874377 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:15:01.881147 28888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c: Bootstrap starting.
I20260620 07:15:01.884927 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.787746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57334 (local address 127.6.53.124:37695)
0620 07:15:01.787942 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.787948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.810879 (+ 22931us) server_negotiation.cc:407] Connection header received
0620 07:15:01.811125 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.811130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.811207 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.811278 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.825233 (+ 13955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.826635 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.856556 (+ 29921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.857421 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.857534 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.875101 (+ 17567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.875109 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.875116 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.875169 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.879421 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.882159 (+ 2738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.882166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.882177 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.882389 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.882976 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.882981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.882985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.883364 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.883644 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.884233 (+ 589us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.884505 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 07:15:01.891855 28888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:01.895007 28888 log.cc:824] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:01.902486 28969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 464193a40c18499e9320463db53a574e. Latest consensus state: current_term: 1 leader_uuid: "464193a40c18499e9320463db53a574e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } } }
I20260620 07:15:01.902890 28969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:15:01.906929 29005 mvcc.cc:205] Tried to move back new op lower bound from 7298825018706325504 to 7298825018389778432. Current Snapshot: MvccSnapshot[applied={T|T < 7298825018706325504}]
I20260620 07:15:01.916488 28983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "464193a40c18499e9320463db53a574e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } } }
I20260620 07:15:01.916932 28983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:01.917256 28995 catalog_manager.cc:1347] Generated new cluster ID: 08d839b1fd1d42149ea97c2ae9c026e6
I20260620 07:15:01.917378 28995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:01.919389 28969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "464193a40c18499e9320463db53a574e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } } }
I20260620 07:15:01.919698 28969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:15:01.933673 28888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c: No bootstrap required, opened a new log
I20260620 07:15:01.948014 28888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } }
I20260620 07:15:01.948439 28888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b47d1691e44e648ed0bcd88acec82c, State: Initialized, Role: FOLLOWER
I20260620 07:15:01.948943 28888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } }
I20260620 07:15:01.952100 28995 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:15:01.954048 28995 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:01.953548 29013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } } }
I20260620 07:15:01.954434 29013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:15:01.954972 28888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:01.955278 28863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "464193a40c18499e9320463db53a574e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "92b47d1691e44e648ed0bcd88acec82c"
I20260620 07:15:01.956821 28862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:01.961551 28862 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c [term 1 FOLLOWER]: Refusing update from remote peer 464193a40c18499e9320463db53a574e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:01.965857 28863 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 464193a40c18499e9320463db53a574e in term 1.
I20260620 07:15:01.972688 28991 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:01.984243 28995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e: Generated new TSK 0
I20260620 07:15:01.985472 28995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:15:02.048436 29017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "464193a40c18499e9320463db53a574e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } } }
I20260620 07:15:02.048882 29017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:15:02.049242 28983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "464193a40c18499e9320463db53a574e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } } }
I20260620 07:15:02.049559 28983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464193a40c18499e9320463db53a574e [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:02.049973 29013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 464193a40c18499e9320463db53a574e. Latest consensus state: current_term: 1 leader_uuid: "464193a40c18499e9320463db53a574e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464193a40c18499e9320463db53a574e" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 44433 } } peers { permanent_uuid: "92b47d1691e44e648ed0bcd88acec82c" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 45329 } } peers { permanent_uuid: "3148417446a84e4b9e8fffb00c9067a3" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 37695 } } }
I20260620 07:15:02.050346 29013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:15:02.096099 29031 catalog_manager.cc:1259] Loaded cluster ID: 08d839b1fd1d42149ea97c2ae9c026e6
I20260620 07:15:02.096208 29031 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c: loading cluster ID for follower catalog manager: success
I20260620 07:15:02.100041 29031 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c: acquiring CA information for follower catalog manager: success
I20260620 07:15:02.102433 29031 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 92b47d1691e44e648ed0bcd88acec82c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 07:15:02.112700 6357 mini_kdc.cc:348] Time spent kdestroy: real 0.106s user 0.000s sys 0.026s
W20260620 07:15:02.168756 28966 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:02.156466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.6.53.126:44433)
0620 07:15:02.156655 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.156663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.166853 (+ 10190us) server_negotiation.cc:407] Connection header received
0620 07:15:02.167153 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.167160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.167247 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.167323 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:02.168508 (+ 1185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37326: BlockingRecv error: recv got EOF from 127.0.0.1:37326 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 07:15:02.170883 28892 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:02.166927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.6.53.125:45329)
0620 07:15:02.167119 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.167125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.169309 (+ 2184us) server_negotiation.cc:407] Connection header received
0620 07:15:02.169544 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.169549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.169652 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.169729 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:02.170648 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60606: BlockingRecv error: recv got EOF from 127.0.0.1:60606 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 07:15:02.181775 28965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:02.179015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.6.53.124:37695)
0620 07:15:02.179222 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.179228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.180205 (+ 977us) server_negotiation.cc:407] Connection header received
0620 07:15:02.180448 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.180453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.180532 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.180599 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:02.181519 (+ 920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57352: BlockingRecv error: recv got EOF from 127.0.0.1:57352 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 07:15:02.246658 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:02.227369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60608 (local address 127.6.53.125:45329)
0620 07:15:02.227546 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.227553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.238891 (+ 11338us) server_negotiation.cc:407] Connection header received
0620 07:15:02.239440 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.239447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.239659 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.239813 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:02.240509 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.241652 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.243594 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.244247 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.244347 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:02.245125 (+ 778us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:02.245803 (+ 678us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:02.245931 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:02.246193 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 07:15:02.246305 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 07:15:02.291141 28965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:02.279578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.6.53.124:37695)
0620 07:15:02.279835 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.279842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.289053 (+ 9211us) server_negotiation.cc:407] Connection header received
0620 07:15:02.289557 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.289562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.289690 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.289772 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:02.290902 (+ 1130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57356: BlockingRecv error: recv got EOF from 127.0.0.1:57356 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 07:15:02.344075 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:02.325584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.6.53.125:45329)
0620 07:15:02.325798 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.325805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.336844 (+ 11039us) server_negotiation.cc:407] Connection header received
0620 07:15:02.337201 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.337207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.337356 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.337434 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:02.338213 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.339438 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.341418 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.342121 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.342223 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:02.343156 (+ 933us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:02.343346 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:02.343409 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:02.343646 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 07:15:02.343733 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 07:15:02.402021 28965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:02.388599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57372 (local address 127.6.53.124:37695)
0620 07:15:02.388793 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.388801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.399506 (+ 10705us) server_negotiation.cc:407] Connection header received
0620 07:15:02.400039 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.400045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.400151 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.400235 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:02.401733 (+ 1498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57372: BlockingRecv error: recv got EOF from 127.0.0.1:57372 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:15:02.423465 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:02.416517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.6.53.125:45329)
0620 07:15:02.416716 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.416722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.417260 (+ 538us) server_negotiation.cc:407] Connection header received
0620 07:15:02.417436 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.417442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.417529 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.417627 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:02.418112 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.419251 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.421270 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.421967 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.422097 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:02.422606 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:02.422746 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:02.422860 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:02.423036 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 07:15:02.423134 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 07:15:02.432777 28965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:02.431044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.6.53.124:37695)
0620 07:15:02.431212 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.431219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.431238 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:15:02.431524 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.431530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.431626 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.431733 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:02.432546 (+ 813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57374: BlockingRecv error: recv got EOF from 127.0.0.1:57374 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:15:02.450294 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:02.443225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60642 (local address 127.6.53.125:45329)
0620 07:15:02.443421 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.443428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.443974 (+ 546us) server_negotiation.cc:407] Connection header received
0620 07:15:02.444182 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.444188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.444279 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.444390 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:02.444891 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.446125 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.448111 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.448854 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.448997 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:02.449464 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:02.449596 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:02.449689 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:02.449872 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 07:15:02.449965 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 07:15:02.460160 28965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:02.457801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57376 (local address 127.6.53.124:37695)
0620 07:15:02.458052 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.458058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.458603 (+ 545us) server_negotiation.cc:407] Connection header received
0620 07:15:02.458760 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.458765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.458871 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.458985 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:02.459899 (+ 914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57376: BlockingRecv error: recv got EOF from 127.0.0.1:57376 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
I20260620 07:15:02.478678 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:02.471964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.6.53.125:45329)
0620 07:15:02.472143 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.472149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.472620 (+ 471us) server_negotiation.cc:407] Connection header received
0620 07:15:02.472777 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.472783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.472893 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.473004 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:02.473412 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.474592 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.476477 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.477196 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.477327 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:02.477792 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:02.477947 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:02.478052 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:02.478209 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 07:15:02.478323 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 07:15:02.493866 28965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:02.491655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.6.53.124:37695)
0620 07:15:02.491823 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.491829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.492321 (+ 492us) server_negotiation.cc:407] Connection header received
0620 07:15:02.492507 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.492513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.492618 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.492737 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:02.493624 (+ 887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57386: BlockingRecv error: recv got EOF from 127.0.0.1:57386 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:15:02.519758 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:02.513110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.6.53.125:45329)
0620 07:15:02.513270 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.513275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.513774 (+ 499us) server_negotiation.cc:407] Connection header received
0620 07:15:02.513953 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.513959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.514052 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.514157 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:02.514639 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.515729 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.517549 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.518294 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.518437 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:02.518975 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:02.519114 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:02.519220 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:02.519379 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 07:15:02.519464 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 07:15:02.527397 28965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:02.525548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57390 (local address 127.6.53.124:37695)
0620 07:15:02.525753 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.525759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.526101 (+ 342us) server_negotiation.cc:407] Connection header received
0620 07:15:02.526215 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.526221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.526357 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.526467 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:02.527150 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57390: BlockingRecv error: recv got EOF from 127.0.0.1:57390 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:15:02.569207 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:02.562565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60684 (local address 127.6.53.125:45329)
0620 07:15:02.562850 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.562859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.563387 (+ 528us) server_negotiation.cc:407] Connection header received
0620 07:15:02.563559 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.563567 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.563659 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.563782 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:02.564149 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.565297 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.567217 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.567813 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.567952 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:02.568376 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:02.568512 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:02.568611 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:02.568780 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 07:15:02.568868 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 07:15:02.577832 28965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:02.575574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.6.53.124:37695)
0620 07:15:02.575754 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.575759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.576334 (+ 575us) server_negotiation.cc:407] Connection header received
0620 07:15:02.576531 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.576537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.576636 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.576736 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:02.577599 (+ 863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57404: BlockingRecv error: recv got EOF from 127.0.0.1:57404 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:15:02.650285 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:02.645131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.6.53.125:45329)
0620 07:15:02.645411 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.645417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.645634 (+ 217us) server_negotiation.cc:407] Connection header received
0620 07:15:02.645860 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.645865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.645939 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.646042 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:02.646539 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.647256 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.648582 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.648998 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.649138 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:02.649520 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:02.649653 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:02.649752 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:02.649906 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 07:15:02.649983 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 07:15:02.657613 28965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:02.655686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57412 (local address 127.6.53.124:37695)
0620 07:15:02.655925 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.655931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.656246 (+ 315us) server_negotiation.cc:407] Connection header received
0620 07:15:02.656453 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.656458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.656571 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.656675 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:02.657363 (+ 688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57412: BlockingRecv error: recv got EOF from 127.0.0.1:57412 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 07:15:02.715847 28980 catalog_manager.cc:1259] Loaded cluster ID: 08d839b1fd1d42149ea97c2ae9c026e6
I20260620 07:15:02.715945 28980 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3: loading cluster ID for follower catalog manager: success
I20260620 07:15:02.718307 28980 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3: acquiring CA information for follower catalog manager: success
I20260620 07:15:02.720036 28980 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3148417446a84e4b9e8fffb00c9067a3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:15:02.794009 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:02.788953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.6.53.125:45329)
0620 07:15:02.789209 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.789217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.789484 (+ 267us) server_negotiation.cc:407] Connection header received
0620 07:15:02.789715 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.789719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.789786 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.789888 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:02.790320 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.791026 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.792342 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.792745 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.792887 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:02.793279 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:02.793391 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:02.793498 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:02.793665 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 07:15:02.793743 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 07:15:02.809690 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:02.804336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57414 (local address 127.6.53.124:37695)
0620 07:15:02.804521 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.804525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.804937 (+ 412us) server_negotiation.cc:407] Connection header received
0620 07:15:02.805053 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.805057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.805201 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.805300 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:02.805659 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.806491 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.807737 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.808151 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.808256 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:02.808737 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:02.809243 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:02.809382 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:02.809423 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 07:15:02.809485 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:15:02.822926 29134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:02.815991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.6.53.126:44433)
0620 07:15:02.816497 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.816504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.816524 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:15:02.816870 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.816878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.817144 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.817352 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:02.818080 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.819137 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.820669 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.821091 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.821219 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:02.821789 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:02.822233 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:02.822367 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:02.822523 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 07:15:02.822611 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":239,"threads_started":1}
I20260620 07:15:02.823901 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:02.816741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.6.53.125:45329)
0620 07:15:02.816941 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.816947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.817169 (+ 222us) server_negotiation.cc:407] Connection header received
0620 07:15:02.817468 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.817476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.817603 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.817676 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:02.818472 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.819504 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.821742 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.822418 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.822550 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:02.823112 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:02.823233 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:02.823317 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:02.823515 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 07:15:02.823611 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:15:02.824136 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:02.817284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.6.53.124:37695)
0620 07:15:02.817456 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:02.817460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:02.818048 (+ 588us) server_negotiation.cc:407] Connection header received
0620 07:15:02.818342 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:02.818347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:02.818424 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:02.818516 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:02.819048 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.820010 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.822009 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:02.822598 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:02.822716 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:02.823384 (+ 668us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:02.823505 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:02.823569 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:02.823783 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 07:15:02.823864 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:15:02.828393 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 28747
I20260620 07:15:02.829562 28809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:15:02.936717 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 28747
I20260620 07:15:02.957695 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 28821
I20260620 07:15:02.958865 28883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:15:03.064883 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 28821
I20260620 07:15:03.084678 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 28895
I20260620 07:15:03.085733 28957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:15:03.190655 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 28895
2026-06-20T07:15:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4404 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:15:03 dist-test-slave-4n6h krb5kdc[29142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:15:03 dist-test-slave-4n6h krb5kdc[29142](info): set up 2 sockets
Jun 20 07:15:03 dist-test-slave-4n6h krb5kdc[29142](info): commencing operation
krb5kdc: starting...
W20260620 07:15:04.314836 6357 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/test-user.keytab.
Jun 20 07:15:04 dist-test-slave-4n6h krb5kdc[29142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:15:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:15:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
Entry for principal kudu/127.6.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab.
I20260620 07:15:04.563736 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:35971
--webserver_interface=127.6.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:42141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:35971,127.6.53.125:35683,127.6.53.124:40527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:04.887728 29155 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:15:04.888093 29155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:04.888223 29155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:04.888366 29155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:04.895468 29155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:04.895602 29155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:04.895707 29155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:04.895794 29155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:04.895879 29155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:04.905164 29155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:42141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:35971,127.6.53.125:35683,127.6.53.124:40527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.126.keytab
--principal=kudu/127.6.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.126:35971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:15:04.906505 29155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:04.907769 29155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:04.919036 29160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:04.918952 29163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:04.918990 29161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:04.920199 29155 server_base.cc:1034] running on GCE node
I20260620 07:15:04.920917 29155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:04.922223 29155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:04.923525 29155 hybrid_clock.cc:648] HybridClock initialized: now 1781939704923464 us; error 43 us; skew 500 ppm
I20260620 07:15:04.923856 29155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:15:04 dist-test-slave-4n6h krb5kdc[29142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939704, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:04.926883 29155 init.cc:377] Logged in from keytab as kudu/127.6.53.126@KRBTEST.COM (short username kudu)
I20260620 07:15:04.928246 29155 webserver.cc:460] Webserver started at http://127.6.53.126:38537/ using document root <none> and password file <none>
I20260620 07:15:04.928936 29155 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:04.929059 29155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:04.929404 29155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:04.932261 29155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/data/instance:
uuid: "1f46fd2fba49466fbbcb88998657f45a"
format_stamp: "Formatted at 2026-06-20 07:15:04 on dist-test-slave-4n6h"
server_key: "8df9da0e8a589fedbaf89b348d09d591"
server_key_iv: "2264ae3ac7ca823f89d7e08733d6b08a"
server_key_version: "encryptionkey@0"
I20260620 07:15:04.933015 29155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/wal/instance:
uuid: "1f46fd2fba49466fbbcb88998657f45a"
format_stamp: "Formatted at 2026-06-20 07:15:04 on dist-test-slave-4n6h"
server_key: "8df9da0e8a589fedbaf89b348d09d591"
server_key_iv: "2264ae3ac7ca823f89d7e08733d6b08a"
server_key_version: "encryptionkey@0"
I20260620 07:15:04.937848 29155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 07:15:04.941232 29170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:04.942665 29155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:04.942842 29155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/wal
uuid: "1f46fd2fba49466fbbcb88998657f45a"
format_stamp: "Formatted at 2026-06-20 07:15:04 on dist-test-slave-4n6h"
server_key: "8df9da0e8a589fedbaf89b348d09d591"
server_key_iv: "2264ae3ac7ca823f89d7e08733d6b08a"
server_key_version: "encryptionkey@0"
I20260620 07:15:04.943054 29155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:04.960436 29155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:04.964443 29155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:04.964723 29155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:04.983098 29155 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.126:35971
I20260620 07:15:04.983117 29221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.126:35971 every 8 connection(s)
I20260620 07:15:04.984448 29155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:04.987965 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 29155
I20260620 07:15:04.988148 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-0/wal/instance
I20260620 07:15:04.988272 29222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:04.988718 6357 external_mini_cluster.cc:1428] Setting key a7d3f024a072b5c790d2b11ea723ffbb
I20260620 07:15:04.998018 29222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:15:05 dist-test-slave-4n6h krb5kdc[29142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939704, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.53.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.125@KRBTEST.COM" created.
I20260620 07:15:05.021014 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.001627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:35971 (local address 127.0.0.1:43012)
0620 07:15:05.002786 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.002822 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:05.004541 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:05.005341 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:05.005347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:05.006030 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:05.006521 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:05.006541 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:05.008070 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:05.008077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:05.009132 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:05.009144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:05.010062 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:05.010069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:05.010296 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:05.011023 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:05.011061 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:05.013781 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:05.017135 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:05.017145 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:05.017153 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:05.017610 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:05.018058 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:05.018062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:05.018064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:05.018222 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:05.019236 (+ 1014us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:05.019242 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:05.019472 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 07:15:05.019664 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 07:15:05.020022 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"thread_start_us":756,"threads_started":1}
I20260620 07:15:05.021051 29225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.001627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.6.53.126:35971)
0620 07:15:05.002535 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:05.002555 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:05.002948 (+ 393us) server_negotiation.cc:407] Connection header received
0620 07:15:05.004893 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:05.004899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:05.005117 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:05.005321 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:05.006731 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:05.007912 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:05.009271 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:05.009823 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:05.010025 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:05.014021 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:05.014028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:05.014040 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:05.014067 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:05.017008 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:05.017758 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:05.017765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:05.017767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:05.017933 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:05.018396 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:05.018401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:05.018404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:05.019097 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:05.019310 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:05.019848 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 07:15:05.020236 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":590,"thread_start_us":385,"threads_started":1}
I20260620 07:15:05.023882 29222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:15:05.024879 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.024496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:35683 (local address 127.0.0.1:49888)
0620 07:15:05.024679 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.024776 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 07:15:05.025157 29172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.53.125:35683: connect: Connection refused (error 111)
I20260620 07:15:05.026728 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.026406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:35683 (local address 127.0.0.1:49892)
0620 07:15:05.026549 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.026612 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 07:15:05.026935 29222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.125:35683: Network error: Client connection negotiation failed: client connection to 127.6.53.125:35683: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.125.keytab.
Entry for principal kudu/127.6.53.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.125.keytab.
I20260620 07:15:05.049603 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.125:35683
--webserver_interface=127.6.53.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.125.keytab
--principal=kudu/127.6.53.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:42141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:35971,127.6.53.125:35683,127.6.53.124:40527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:15:05.076228 29222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } attempt: 1
I20260620 07:15:05.077334 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.076886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:35683 (local address 127.0.0.1:49902)
0620 07:15:05.077113 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.077196 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 07:15:05.078498 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.078102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:35683 (local address 127.0.0.1:49916)
0620 07:15:05.078287 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.078358 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 07:15:05.078713 29222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.125:35683: Network error: Client connection negotiation failed: client connection to 127.6.53.125:35683: connect: Connection refused (error 111)
I20260620 07:15:05.146973 29222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } attempt: 2
I20260620 07:15:05.148156 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.147626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:35683 (local address 127.0.0.1:49920)
0620 07:15:05.147908 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.148002 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 07:15:05.149205 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.148862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:35683 (local address 127.0.0.1:49930)
0620 07:15:05.149011 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.149083 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 07:15:05.149433 29222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.125:35683: Network error: Client connection negotiation failed: client connection to 127.6.53.125:35683: connect: Connection refused (error 111)
I20260620 07:15:05.240689 29222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } attempt: 3
I20260620 07:15:05.241860 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.241440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:35683 (local address 127.0.0.1:49942)
0620 07:15:05.241637 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.241728 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 07:15:05.242908 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.242567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:35683 (local address 127.0.0.1:49958)
0620 07:15:05.242707 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.242777 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 07:15:05.243088 29222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.125:35683: Network error: Client connection negotiation failed: client connection to 127.6.53.125:35683: connect: Connection refused (error 111)
W20260620 07:15:05.369746 29229 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:15:05.370086 29229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:05.370222 29229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:05.370397 29229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:05.377662 29229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:05.377754 29229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:05.377851 29229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:05.377947 29229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:05.378057 29229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:05.386318 29222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } attempt: 4
I20260620 07:15:05.387729 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.387104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:35683 (local address 127.0.0.1:49964)
0620 07:15:05.387409 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.387525 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260620 07:15:05.388808 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.388489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:35683 (local address 127.0.0.1:49966)
0620 07:15:05.388627 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.388699 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.125:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 07:15:05.389045 29222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.125:35683: Network error: Client connection negotiation failed: client connection to 127.6.53.125:35683: connect: Connection refused (error 111)
I20260620 07:15:05.393536 29229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:42141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:35971,127.6.53.125:35683,127.6.53.124:40527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.125.keytab
--principal=kudu/127.6.53.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.125:35683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.53.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:15:05.395174 29229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:05.397142 29229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:05.411900 29234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:05.412488 29237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:05.412592 29235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:05.413112 29229 server_base.cc:1034] running on GCE node
I20260620 07:15:05.414089 29229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:05.415777 29229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:05.417018 29229 hybrid_clock.cc:648] HybridClock initialized: now 1781939705416905 us; error 77 us; skew 500 ppm
I20260620 07:15:05.417387 29229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:15:05 dist-test-slave-4n6h krb5kdc[29142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939705, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:05.421664 29229 init.cc:377] Logged in from keytab as kudu/127.6.53.125@KRBTEST.COM (short username kudu)
I20260620 07:15:05.423381 29229 webserver.cc:460] Webserver started at http://127.6.53.125:33321/ using document root <none> and password file <none>
I20260620 07:15:05.424247 29229 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:05.424355 29229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:05.424681 29229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:05.429260 29229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/data/instance:
uuid: "867f1e5a0ed146b581bcc6a4f23d165e"
format_stamp: "Formatted at 2026-06-20 07:15:05 on dist-test-slave-4n6h"
server_key: "6cd56e730440b7837ae1125bcad57345"
server_key_iv: "10551917876f22c2a60ce49bde89add3"
server_key_version: "encryptionkey@0"
I20260620 07:15:05.430462 29229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/wal/instance:
uuid: "867f1e5a0ed146b581bcc6a4f23d165e"
format_stamp: "Formatted at 2026-06-20 07:15:05 on dist-test-slave-4n6h"
server_key: "6cd56e730440b7837ae1125bcad57345"
server_key_iv: "10551917876f22c2a60ce49bde89add3"
server_key_version: "encryptionkey@0"
I20260620 07:15:05.437223 29229 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 07:15:05.440647 29244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:05.442121 29229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:15:05.442322 29229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/wal
uuid: "867f1e5a0ed146b581bcc6a4f23d165e"
format_stamp: "Formatted at 2026-06-20 07:15:05 on dist-test-slave-4n6h"
server_key: "6cd56e730440b7837ae1125bcad57345"
server_key_iv: "10551917876f22c2a60ce49bde89add3"
server_key_version: "encryptionkey@0"
I20260620 07:15:05.442538 29229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:05.466138 29229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:05.470012 29229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:05.470223 29229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:05.488270 29229 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.125:35683
I20260620 07:15:05.488289 29295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.125:35683 every 8 connection(s)
I20260620 07:15:05.489578 29229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/data/info.pb
I20260620 07:15:05.493119 29296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:05.493939 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 29229
I20260620 07:15:05.494133 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-1/wal/instance
I20260620 07:15:05.494714 6357 external_mini_cluster.cc:1428] Setting key 46ff44592e6a9da950cb3871e0ff596f
I20260620 07:15:05.502910 29296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:15:05 dist-test-slave-4n6h krb5kdc[29142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939705, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.125@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.53.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.53.124@KRBTEST.COM" created.
I20260620 07:15:05.523622 29225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.506137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.6.53.126:35971)
0620 07:15:05.506387 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:05.506393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:05.507634 (+ 1241us) server_negotiation.cc:407] Connection header received
0620 07:15:05.509422 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:05.509430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:05.509522 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:05.509642 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:05.511254 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:05.512380 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:05.513884 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:05.514371 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:05.514513 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:05.518078 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:05.518083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:05.518086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:05.518118 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:05.520736 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:05.521438 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:05.521444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:05.521448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:05.521566 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:05.521978 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:05.521984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:05.521987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:05.522303 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:05.522455 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:05.523081 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 07:15:05.523306 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 07:15:05.524531 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.506631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:35971 (local address 127.0.0.1:43024)
0620 07:15:05.507449 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.507499 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:05.509155 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:05.509818 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:05.509825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:05.510405 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:05.511066 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:05.511085 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:05.512545 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:05.512549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:05.513718 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:05.513730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:05.514582 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:05.514590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:05.514812 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:05.515509 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:05.515529 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:05.517855 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:05.520882 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:05.520891 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:05.520906 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:05.521307 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:05.521684 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:05.521689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:05.521692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:05.521869 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:05.522427 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:05.522435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:05.522751 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 07:15:05.523000 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 07:15:05.523483 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":347,"threads_started":1}
I20260620 07:15:05.526062 29296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:15:05 dist-test-slave-4n6h krb5kdc[29142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939705, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.125@KRBTEST.COM for kudu/127.6.53.125@KRBTEST.COM
I20260620 07:15:05.540478 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.526713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:35683 (local address 127.0.0.1:49970)
0620 07:15:05.526938 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.526952 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:05.527047 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:05.527886 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:05.527891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:05.528280 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:05.528403 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:05.528410 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:05.529700 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:05.529705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:05.530584 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:05.530595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:05.531705 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:05.531711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:05.532001 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:05.532589 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:05.532606 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:05.534881 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:05.538032 (+ 3151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:05.538039 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:05.538044 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:05.538472 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:05.538822 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:05.538829 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:05.538832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:05.538963 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:05.539540 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:05.539544 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:05.539672 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 07:15:05.539811 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 07:15:05.540081 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":48}
I20260620 07:15:05.540558 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.526965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.6.53.125:35683)
0620 07:15:05.527528 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:05.527534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:05.527557 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:15:05.527595 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:05.527600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:05.527750 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:05.527902 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:05.528569 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:05.529570 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:05.530890 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:05.531537 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:05.531711 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:05.535100 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:05.535112 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:05.535123 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:05.535157 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:05.537897 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:05.538593 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:05.538600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:05.538604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:05.538719 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:05.539070 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:05.539075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:05.539078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:05.539413 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:05.539604 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:05.539953 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 07:15:05.540185 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":435,"thread_start_us":306,"threads_started":1}
I20260620 07:15:05.542922 29296 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:15:05.544243 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.543628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33636)
0620 07:15:05.543840 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.544055 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 07:15:05.544610 29245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
I20260620 07:15:05.546305 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.545831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33642)
0620 07:15:05.546079 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.546147 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260620 07:15:05.546563 29296 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:40527: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.53.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.124.keytab.
Entry for principal kudu/127.6.53.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.124.keytab.
I20260620 07:15:05.554980 6357 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
/tmp/dist-test-taskS8B4yM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.124:40527
--webserver_interface=127.6.53.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.124.keytab
--principal=kudu/127.6.53.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.53.84:42141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.53.126:35971,127.6.53.125:35683,127.6.53.124:40527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:15:05.595930 29296 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } attempt: 1
I20260620 07:15:05.597074 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.596631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33644)
0620 07:15:05.596817 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.596932 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 07:15:05.598044 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.597695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33646)
0620 07:15:05.597868 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.597936 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 07:15:05.598325 29296 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:40527: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
I20260620 07:15:05.666565 29296 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } attempt: 2
I20260620 07:15:05.667742 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.667224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33656)
0620 07:15:05.667444 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.667554 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 07:15:05.668774 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.668397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33660)
0620 07:15:05.668548 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.668633 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 07:15:05.668988 29296 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:40527: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
I20260620 07:15:05.688378 29222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } attempt: 5
Jun 20 07:15:05 dist-test-slave-4n6h krb5kdc[29142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939704, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for kudu/127.6.53.125@KRBTEST.COM
I20260620 07:15:05.704015 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.689215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:35683 (local address 127.0.0.1:49984)
0620 07:15:05.689499 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.689541 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:05.689684 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:05.690106 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:05.690111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:05.690562 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:05.690733 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:05.690742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:05.692161 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:05.692165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:05.693040 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:05.693050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:05.694187 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:05.694192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:05.694500 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:05.695179 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:05.695197 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:05.697446 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:05.701567 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:05.701571 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:05.701574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:05.701912 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:05.702340 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:05.702343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:05.702345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:05.702507 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:05.703127 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:05.703131 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:05.703261 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 07:15:05.703436 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 07:15:05.703736 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260620 07:15:05.704105 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.689347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49984 (local address 127.6.53.125:35683)
0620 07:15:05.689597 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:05.689604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:05.689629 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:15:05.689821 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:05.689827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:05.689918 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:05.690028 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:05.690913 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:05.692013 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:05.693203 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:05.694019 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:05.694176 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:05.697636 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:05.697643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:05.697648 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:05.697686 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:05.701356 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:05.702028 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:05.702034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:05.702037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:05.702179 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:05.702614 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:05.702619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:05.702622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:05.702985 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:05.703129 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:05.703571 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 07:15:05.703787 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:15:05.705322 29222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:15:05.706416 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.705948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33670)
0620 07:15:05.706121 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.706205 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 07:15:05.707391 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.707082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33686)
0620 07:15:05.707208 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.707269 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 07:15:05.707680 29222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:40527: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
I20260620 07:15:05.758921 29222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } attempt: 1
I20260620 07:15:05.760130 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.759675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33690)
0620 07:15:05.759878 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.759971 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 07:15:05.760233 29296 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } attempt: 3
I20260620 07:15:05.761171 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.760824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33710)
0620 07:15:05.761001 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.761064 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 07:15:05.761191 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.760734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33696)
0620 07:15:05.760934 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.761025 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 07:15:05.761399 29222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:40527: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
I20260620 07:15:05.762323 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.761810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33726)
0620 07:15:05.762054 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.762143 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260620 07:15:05.762538 29296 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:40527: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
I20260620 07:15:05.829643 29222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } attempt: 2
I20260620 07:15:05.830984 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.830409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33742)
0620 07:15:05.830652 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.830782 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 07:15:05.832034 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.831633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33744)
0620 07:15:05.831801 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.831898 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 07:15:05.832252 29222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:40527: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
W20260620 07:15:05.881246 29303 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:15:05.881549 29303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:05.881632 29303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:05.881711 29303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:05.888710 29303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:05.888787 29303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:05.888842 29303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:05.888892 29303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:05.888954 29303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:05.898212 29303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.53.84:42141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.53.126:35971,127.6.53.125:35683,127.6.53.124:40527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/krb5kdc/kudu_127.6.53.124.keytab
--principal=kudu/127.6.53.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.53.124:40527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.53.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:15:05.899363 29303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:05.900554 29303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:15:05.905772 29296 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } attempt: 4
I20260620 07:15:05.906983 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.906446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33748)
0620 07:15:05.906669 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.906780 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 07:15:05.908025 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.907732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33764)
0620 07:15:05.907865 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.907931 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 07:15:05.908275 29296 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:40527: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
W20260620 07:15:05.912827 29308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:05.912772 29309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:05.913075 29311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:05.913722 29303 server_base.cc:1034] running on GCE node
I20260620 07:15:05.914505 29303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:05.915859 29303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:05.917097 29303 hybrid_clock.cc:648] HybridClock initialized: now 1781939705917035 us; error 39 us; skew 500 ppm
I20260620 07:15:05.917411 29303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:15:05 dist-test-slave-4n6h krb5kdc[29142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939705, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:05.920428 29303 init.cc:377] Logged in from keytab as kudu/127.6.53.124@KRBTEST.COM (short username kudu)
I20260620 07:15:05.921744 29303 webserver.cc:460] Webserver started at http://127.6.53.124:45627/ using document root <none> and password file <none>
I20260620 07:15:05.922472 29303 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:05.922600 29303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:05.922921 29303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:05.925653 29303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/data/instance:
uuid: "f549209beba94506bd36d9f91c02e52e"
format_stamp: "Formatted at 2026-06-20 07:15:05 on dist-test-slave-4n6h"
server_key: "a6d38e7cf8abed6a077744a1733127c3"
server_key_iv: "e077d92b04189638fd73bf2bcabb2a67"
server_key_version: "encryptionkey@0"
I20260620 07:15:05.926484 29303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/wal/instance:
uuid: "f549209beba94506bd36d9f91c02e52e"
format_stamp: "Formatted at 2026-06-20 07:15:05 on dist-test-slave-4n6h"
server_key: "a6d38e7cf8abed6a077744a1733127c3"
server_key_iv: "e077d92b04189638fd73bf2bcabb2a67"
server_key_version: "encryptionkey@0"
I20260620 07:15:05.931265 29303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:15:05.934489 29318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:05.935706 29303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:15:05.935897 29303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/data,/tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/wal
uuid: "f549209beba94506bd36d9f91c02e52e"
format_stamp: "Formatted at 2026-06-20 07:15:05 on dist-test-slave-4n6h"
server_key: "a6d38e7cf8abed6a077744a1733127c3"
server_key_iv: "e077d92b04189638fd73bf2bcabb2a67"
server_key_version: "encryptionkey@0"
I20260620 07:15:05.936108 29303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:05.938454 29222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } attempt: 3
I20260620 07:15:05.939435 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.939036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33766)
0620 07:15:05.939210 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.939290 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 07:15:05.940392 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.940035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33776)
0620 07:15:05.940228 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.940292 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 07:15:05.940595 29222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.53.124:40527: Network error: Client connection negotiation failed: client connection to 127.6.53.124:40527: connect: Connection refused (error 111)
I20260620 07:15:05.956557 29303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:05.960837 29303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:05.961123 29303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:05.979638 29303 rpc_server.cc:307] RPC server started. Bound to: 127.6.53.124:40527
I20260620 07:15:05.979656 29369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.53.124:40527 every 8 connection(s)
I20260620 07:15:05.980952 29303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/data/info.pb
I20260620 07:15:05.985030 29370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:05.988349 6357 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu as pid 29303
I20260620 07:15:05.988533 6357 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS8B4yM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939352012798-6357-0/minicluster-data/master-2/wal/instance
I20260620 07:15:05.989029 6357 external_mini_cluster.cc:1428] Setting key 8cf9a456d281c7402d5d6e8b591b0de9
I20260620 07:15:05.994091 29370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:15:06 dist-test-slave-4n6h krb5kdc[29142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939705, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.124@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
Jun 20 07:15:06 dist-test-slave-4n6h krb5kdc[29142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.124@KRBTEST.COM
I20260620 07:15:06.014243 29225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.998157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.6.53.126:35971)
0620 07:15:05.998347 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:05.998352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:05.999136 (+ 784us) server_negotiation.cc:407] Connection header received
0620 07:15:05.999324 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:05.999328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:05.999392 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:05.999467 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.000992 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.001708 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.003613 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.004061 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.004193 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.007488 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:06.007495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:06.007500 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:06.007540 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:06.010912 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.011694 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.011699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.011702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.011886 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.012365 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.012369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.012371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.012738 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:06.012870 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.013623 (+ 753us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.013835 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:15:06.015220 29373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.990946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.6.53.124:40527)
0620 07:15:05.993785 (+ 2839us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:05.993800 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:05.994285 (+ 485us) server_negotiation.cc:407] Connection header received
0620 07:15:05.996194 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:05.996220 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:05.996696 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:05.997089 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:05.998785 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.000809 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.002561 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.003352 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.003468 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.008073 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:06.008080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:06.008091 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:06.008117 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:06.011722 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.012519 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.012527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.012530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.012671 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.013110 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.013113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.013115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.013415 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:06.013729 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.014055 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.014330 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2594,"thread_start_us":2335,"threads_started":1}
I20260620 07:15:06.015268 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:05.998210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.126:35971 (local address 127.0.0.1:43028)
0620 07:15:05.998961 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:05.999016 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:05.999185 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:05.999558 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:05.999565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:06.000419 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:06.000792 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:06.000808 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.001919 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.001926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:06.003468 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:06.003481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.004272 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.004280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:06.004449 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.005051 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:06.005081 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:06.007284 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:06.011034 (+ 3750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:06.011042 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:06.011051 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:06.011565 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:06.011998 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:06.012003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:06.012007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:06.012223 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:06.012871 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:06.012879 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:06.013271 (+ 392us) client_negotiation.cc:769] Sending connection context
0620 07:15:06.013495 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 07:15:06.014005 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":336,"threads_started":1}
I20260620 07:15:06.016839 29370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:15:06 dist-test-slave-4n6h krb5kdc[29142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939705, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.124@KRBTEST.COM for kudu/127.6.53.125@KRBTEST.COM
Jun 20 07:15:06 dist-test-slave-4n6h krb5kdc[29142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.126@KRBTEST.COM
I20260620 07:15:06.031701 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.017495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.125:35683 (local address 127.0.0.1:49986)
0620 07:15:06.017832 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:06.017851 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:06.017961 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:06.018344 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:06.018352 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:06.018802 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:06.018980 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:06.018992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.020439 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.020444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:06.021782 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:06.021793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.022730 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.022734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:06.022867 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.023405 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:06.023421 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:06.025821 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:06.029364 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:06.029368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:06.029370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:06.029714 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:06.030113 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:06.030119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:06.030122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:06.030268 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:06.030808 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:06.030813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:06.030952 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 07:15:06.031094 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 07:15:06.031353 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 07:15:06.031756 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.017585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.6.53.125:35683)
0620 07:15:06.017754 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.017760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.017933 (+ 173us) server_negotiation.cc:407] Connection header received
0620 07:15:06.018081 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.018086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.018163 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.018330 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.019133 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.020260 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.021907 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.022599 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.022721 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.025998 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:06.026003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:06.026007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:06.026048 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:06.029237 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.029861 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.029867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.029870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.030002 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.030428 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.030432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.030434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.030683 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:06.030832 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.031191 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.031397 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:15:06.032729 29370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:15:06.033825 29225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.018676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.6.53.126:35971)
0620 07:15:06.018849 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.018857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.018875 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:15:06.018934 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.018941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.019007 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.019097 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.020463 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.021561 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.023192 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.023597 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.023730 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.028069 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:06.028077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:06.028085 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:06.028122 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:06.031119 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.031753 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.031760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.031765 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.031932 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.032361 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.032365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.032367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.032690 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:06.032928 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.033234 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.033463 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 07:15:06 dist-test-slave-4n6h krb5kdc[29142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939705, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.124@KRBTEST.COM for kudu/127.6.53.124@KRBTEST.COM
Jun 20 07:15:06 dist-test-slave-4n6h krb5kdc[29142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.53.125@KRBTEST.COM
I20260620 07:15:06.046067 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.033603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33796)
0620 07:15:06.033759 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:06.033773 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:06.033840 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:06.034196 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:06.034199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:06.034599 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:06.034711 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:06.034719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.036033 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.036036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:06.036860 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:06.036870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.037534 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.037541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:06.037757 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.038367 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:06.038398 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:06.040640 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:06.043728 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:06.043735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:06.043738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:06.044097 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:06.044453 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:06.044459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:06.044464 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:06.044583 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:06.045050 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:06.045055 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:06.045191 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 07:15:06.045333 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 07:15:06.045623 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 07:15:06.046151 29373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.033818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33796 (local address 127.6.53.124:40527)
0620 07:15:06.033984 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.033989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.034010 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:15:06.034040 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.034043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.034086 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.034183 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.034842 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.035834 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.037023 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.037394 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.037489 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.040805 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:06.040828 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:06.040836 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:06.040892 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:06.043600 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.044213 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.044220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.044224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.044348 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.044684 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.044691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.044695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.044942 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:06.045087 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.045427 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.045811 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":64}
I20260620 07:15:06.049692 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.035458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.6.53.125:35683)
0620 07:15:06.035610 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.035614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.035646 (+ 32us) server_negotiation.cc:407] Connection header received
0620 07:15:06.035680 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.035683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.035729 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.035791 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.036972 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.037807 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.039069 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.039597 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.039687 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.043780 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:06.043785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:06.043788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:06.043816 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:06.047266 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.047997 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.048001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.048003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.048130 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.048540 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.048543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.048545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.048785 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:06.048996 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.049284 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.049442 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:15:06.052255 29370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e: Bootstrap starting.
I20260620 07:15:06.054338 6357 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:15:06.056087 29370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:06.057327 29370 log.cc:824] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:06.057305 29197 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1f46fd2fba49466fbbcb88998657f45a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43044
I20260620 07:15:06.060067 29370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e: No bootstrap required, opened a new log
I20260620 07:15:06.066921 29370 raft_consensus.cc:348] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } }
I20260620 07:15:06.067225 29370 raft_consensus.cc:729] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f549209beba94506bd36d9f91c02e52e, State: Initialized, Role: FOLLOWER
I20260620 07:15:06.067648 29370 consensus_queue.cc:260] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } }
I20260620 07:15:06.068445 29376 sys_catalog.cc:455] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } } }
I20260620 07:15:06.068778 29376 sys_catalog.cc:458] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:15:06.069641 29370 sys_catalog.cc:564] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e [sys.catalog]: configured and running, proceeding with master startup.
W20260620 07:15:06.078961 29387 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:15:06.079092 29387 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:15:06.076784 29303 master_runner.cc:186] Error getting master registration for 127.6.53.126:35971: OK, instance_id { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" instance_seqno: 1781939704978000 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:15:06.087972 29303 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:15:06.117795 29222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } attempt: 4
Jun 20 07:15:06 dist-test-slave-4n6h krb5kdc[29142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939704, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.126@KRBTEST.COM for kudu/127.6.53.124@KRBTEST.COM
I20260620 07:15:06.133929 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.118572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33804)
0620 07:15:06.118779 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:06.118801 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:06.118946 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:06.119292 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:06.119299 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:06.119797 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:06.120003 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:06.120015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.121540 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.121545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:06.123087 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:06.123100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.124135 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.124142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:06.124336 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.124998 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:06.125016 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:06.127145 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:06.131083 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:06.131091 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:06.131095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:06.131600 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:06.131997 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:06.132002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:06.132005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:06.132194 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:06.132902 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:06.132907 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:06.133067 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 07:15:06.133248 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 07:15:06.133525 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 07:15:06.133988 29373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.118746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33804 (local address 127.6.53.124:40527)
0620 07:15:06.118950 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.118956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.118979 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:15:06.119056 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.119066 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.119144 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.119253 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.120166 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.121370 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.123240 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.123985 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.124113 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.127333 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:06.127339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:06.127344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:06.127381 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:06.130947 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.131720 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.131725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.131729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.131890 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.132301 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.132306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.132310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.132747 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:06.132948 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.133364 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.133603 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:15:06.138501 29222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a: Bootstrap starting.
I20260620 07:15:06.142202 29222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:06.143276 29222 log.cc:824] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:06.146090 29222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a: No bootstrap required, opened a new log
I20260620 07:15:06.151435 29222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } }
I20260620 07:15:06.151746 29222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f46fd2fba49466fbbcb88998657f45a, State: Initialized, Role: FOLLOWER
I20260620 07:15:06.152177 29222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } }
I20260620 07:15:06.153086 29390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } } }
I20260620 07:15:06.153494 29390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:15:06.154136 29222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:06.154276 29197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 07:15:06.154466 29197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:06.158004 29197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } }
I20260620 07:15:06.160772 29197 leader_election.cc:288] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [CANDIDATE]: Term 1 election: Requested vote from peers 867f1e5a0ed146b581bcc6a4f23d165e (127.6.53.125:35683), f549209beba94506bd36d9f91c02e52e (127.6.53.124:40527)
I20260620 07:15:06.163700 29345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f46fd2fba49466fbbcb88998657f45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f549209beba94506bd36d9f91c02e52e"
W20260620 07:15:06.164026 29401 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:15:06.164152 29401 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:15:06.164253 29345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:06.168583 29345 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f46fd2fba49466fbbcb88998657f45a in term 1.
I20260620 07:15:06.174651 29171 leader_election.cc:302] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f46fd2fba49466fbbcb88998657f45a, f549209beba94506bd36d9f91c02e52e; no voters:
I20260620 07:15:06.175168 29390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:06.186363 29390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [term 1 LEADER]: Becoming Leader. State: Replica: 1f46fd2fba49466fbbcb88998657f45a, State: Running, Role: LEADER
I20260620 07:15:06.187124 29390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } }
I20260620 07:15:06.170327 29155 master_runner.cc:186] Error getting master registration for 127.6.53.125:35683: OK, instance_id { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" instance_seqno: 1781939705483254 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:15:06.197491 29155 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:15:06.206821 29405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f46fd2fba49466fbbcb88998657f45a. Latest consensus state: current_term: 1 leader_uuid: "1f46fd2fba49466fbbcb88998657f45a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } } }
I20260620 07:15:06.207264 29405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:06.207526 29296 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } attempt: 5
W20260620 07:15:06.213398 29324 tablet.cc:2369] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:15:06.217496 29176 tablet.cc:2369] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 07:15:06 dist-test-slave-4n6h krb5kdc[29142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939705, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.53.125@KRBTEST.COM for kudu/127.6.53.124@KRBTEST.COM
I20260620 07:15:06.242041 29409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:06.244221 29409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:06.245690 29373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.208734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.6.53.124:40527)
0620 07:15:06.208917 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.208923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.208941 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:15:06.209003 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.209008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.209081 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.209190 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.210413 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.211584 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.213939 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.214656 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.214775 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.232288 (+ 17513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:06.232300 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:06.232308 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:06.232372 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:06.237351 (+ 4979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.240034 (+ 2683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.240074 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.240081 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.240503 (+ 422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.241463 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.241503 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.241508 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.242489 (+ 981us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:06.242817 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.243886 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.244600 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:15:06.248399 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.208270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.53.124:40527 (local address 127.0.0.1:33806)
0620 07:15:06.208532 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:06.208559 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:06.208723 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:06.209276 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:06.209283 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:06.209914 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:06.210208 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:06.210226 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.211732 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.211740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:06.213764 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:06.213785 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.214868 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.214876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:06.215261 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.216080 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:06.216128 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:06.231583 (+ 15455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:06.237626 (+ 6043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:06.237635 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:06.237642 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:06.239729 (+ 2087us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:06.240801 (+ 1072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:06.240833 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:06.240839 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:06.241224 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:06.242969 (+ 1745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:06.242977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:06.243312 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 07:15:06.246700 (+ 3388us) client_negotiation.cc:240] Negotiation successful
0620 07:15:06.247430 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 07:15:06.256266 29296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e: Bootstrap starting.
I20260620 07:15:06.265923 29296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:06.268038 29296 log.cc:824] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:06.286221 29225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.217385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43052 (local address 127.6.53.126:35971)
0620 07:15:06.217585 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.217592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.233872 (+ 16280us) server_negotiation.cc:407] Connection header received
0620 07:15:06.236431 (+ 2559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.236437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.236524 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.236595 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.249412 (+ 12817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.250499 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.261047 (+ 10548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.261736 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.261877 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.272257 (+ 10380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:06.272280 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:06.272285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:06.272331 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:06.276191 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.281999 (+ 5808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.282005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.282009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.282273 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.284263 (+ 1990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.284269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.284273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.284700 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:06.284948 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.285513 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.285774 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:15:06.295962 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.216982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33814 (local address 127.6.53.124:40527)
0620 07:15:06.239397 (+ 22415us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.239403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.239427 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:15:06.239474 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.239479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.239553 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.239651 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.247912 (+ 8261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.249212 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.260213 (+ 11001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.260905 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.266977 (+ 6072us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.277830 (+ 10853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:06.277838 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:06.277843 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:06.277916 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:06.281621 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.290752 (+ 9131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.290758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.290762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.290993 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.293557 (+ 2564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.293563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.293567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.293985 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:06.294269 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.295283 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.295525 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22257,"thread_start_us":22096,"threads_started":1}
I20260620 07:15:06.306828 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.234564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50002 (local address 127.6.53.125:35683)
0620 07:15:06.234739 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.234744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.262329 (+ 27585us) server_negotiation.cc:407] Connection header received
0620 07:15:06.262969 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.262979 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.263060 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.263151 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.269867 (+ 6716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.271032 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.288804 (+ 17772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.289537 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.289640 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.299833 (+ 10193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:06.299839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:06.299843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:06.299898 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:06.303601 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.304392 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.304397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.304401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.304558 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:06.304995 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:06.305001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:06.305004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:06.305372 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:06.305602 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.306200 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.306471 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:15:06.310320 29345 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e [term 1 FOLLOWER]: Refusing update from remote peer 1f46fd2fba49466fbbcb88998657f45a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:06.311393 29405 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:06.314494 29296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e: No bootstrap required, opened a new log
I20260620 07:15:06.331393 29296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } }
I20260620 07:15:06.332036 29296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 867f1e5a0ed146b581bcc6a4f23d165e, State: Initialized, Role: FOLLOWER
I20260620 07:15:06.332768 29296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } }
I20260620 07:15:06.336983 29376 sys_catalog.cc:455] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f46fd2fba49466fbbcb88998657f45a. Latest consensus state: current_term: 1 leader_uuid: "1f46fd2fba49466fbbcb88998657f45a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } } }
I20260620 07:15:06.337285 29376 sys_catalog.cc:458] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:15:06.341733 29414 mvcc.cc:205] Tried to move back new op lower bound from 7298825037000462336 to 7298825036549656576. Current Snapshot: MvccSnapshot[applied={T|T < 7298825037000462336}]
I20260620 07:15:06.345988 29416 mvcc.cc:205] Tried to move back new op lower bound from 7298825037000462336 to 7298825036549656576. Current Snapshot: MvccSnapshot[applied={T|T < 7298825037000462336}]
I20260620 07:15:06.356189 29390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f46fd2fba49466fbbcb88998657f45a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } } }
I20260620 07:15:06.356529 29390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:06.359539 29409 catalog_manager.cc:1347] Generated new cluster ID: a74cce3e758c448a9645731d6e8d9156
I20260620 07:15:06.359651 29409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:06.362951 29418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } } }
I20260620 07:15:06.363255 29376 sys_catalog.cc:455] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f46fd2fba49466fbbcb88998657f45a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } } }
I20260620 07:15:06.363619 29418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:15:06.363672 29376 sys_catalog.cc:458] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:15:06.364169 29296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:06.364244 29271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f46fd2fba49466fbbcb88998657f45a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "867f1e5a0ed146b581bcc6a4f23d165e"
I20260620 07:15:06.364678 29271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:06.368254 29271 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f46fd2fba49466fbbcb88998657f45a in term 1.
I20260620 07:15:06.369153 29270 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e [term 1 FOLLOWER]: Refusing update from remote peer 1f46fd2fba49466fbbcb88998657f45a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:06.373888 29405 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [LEADER]: Connected to new peer: Peer: permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:06.408463 29418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f46fd2fba49466fbbcb88998657f45a. Latest consensus state: current_term: 1 leader_uuid: "1f46fd2fba49466fbbcb88998657f45a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } } }
I20260620 07:15:06.408901 29418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:15:06.410580 29427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f46fd2fba49466fbbcb88998657f45a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } } }
I20260620 07:15:06.411123 29427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:15:06.417811 29405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f46fd2fba49466fbbcb88998657f45a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f46fd2fba49466fbbcb88998657f45a" member_type: VOTER last_known_addr { host: "127.6.53.126" port: 35971 } } peers { permanent_uuid: "867f1e5a0ed146b581bcc6a4f23d165e" member_type: VOTER last_known_addr { host: "127.6.53.125" port: 35683 } } peers { permanent_uuid: "f549209beba94506bd36d9f91c02e52e" member_type: VOTER last_known_addr { host: "127.6.53.124" port: 40527 } } }
I20260620 07:15:06.418150 29405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:06.429634 29409 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:15:06.431563 29409 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:06.454144 29409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f46fd2fba49466fbbcb88998657f45a: Generated new TSK 0
I20260620 07:15:06.455199 29409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:15:06.460937 29435 catalog_manager.cc:1259] Loaded cluster ID: a74cce3e758c448a9645731d6e8d9156
I20260620 07:15:06.461102 29435 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e: loading cluster ID for follower catalog manager: success
I20260620 07:15:06.464977 29435 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e: acquiring CA information for follower catalog manager: success
I20260620 07:15:06.470263 29435 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 867f1e5a0ed146b581bcc6a4f23d165e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 07:15:06.620702 29225 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:06.596253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.6.53.126:35971)
0620 07:15:06.596454 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.596460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.618536 (+ 22076us) server_negotiation.cc:407] Connection header received
0620 07:15:06.619057 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.619064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.619176 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.619251 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.620446 (+ 1195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43058: BlockingRecv error: recv got EOF from 127.0.0.1:43058 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 07:15:06.623325 29410 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:06.602955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.6.53.124:40527)
0620 07:15:06.603152 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.603157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.621506 (+ 18349us) server_negotiation.cc:407] Connection header received
0620 07:15:06.622020 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.622026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.622115 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.622188 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.623069 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33826: BlockingRecv error: recv got EOF from 127.0.0.1:33826 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 07:15:06.626842 29300 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:06.624508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.6.53.125:35683)
0620 07:15:06.624698 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.624704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.625273 (+ 569us) server_negotiation.cc:407] Connection header received
0620 07:15:06.625534 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.625541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.625623 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.625692 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.626605 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50008: BlockingRecv error: recv got EOF from 127.0.0.1:50008 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:15:06.678777 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.659736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.6.53.125:35683)
0620 07:15:06.659931 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.659938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.670606 (+ 10668us) server_negotiation.cc:407] Connection header received
0620 07:15:06.671200 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.671207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.671460 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.671607 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:06.672364 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.673577 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.675706 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.676402 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.676497 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.677255 (+ 758us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:06.677921 (+ 666us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:06.678046 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.678322 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.678419 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 07:15:06.709112 29410 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:06.706523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.6.53.124:40527)
0620 07:15:06.706709 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.706716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.707231 (+ 515us) server_negotiation.cc:407] Connection header received
0620 07:15:06.707494 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.707500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.707635 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.707712 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.708851 (+ 1139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33828: BlockingRecv error: recv got EOF from 127.0.0.1:33828 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:15:06.723606 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.713604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50016 (local address 127.6.53.125:35683)
0620 07:15:06.713838 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.713844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.717322 (+ 3478us) server_negotiation.cc:407] Connection header received
0620 07:15:06.717551 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.717557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.717647 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.717753 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:06.718293 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.719471 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.721389 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.722076 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.722210 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.722769 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:06.722941 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:06.723046 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.723215 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.723296 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 07:15:06.736366 29410 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:06.731241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.6.53.124:40527)
0620 07:15:06.731449 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.731455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.734809 (+ 3354us) server_negotiation.cc:407] Connection header received
0620 07:15:06.735012 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.735017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.735113 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.735218 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.736128 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33832: BlockingRecv error: recv got EOF from 127.0.0.1:33832 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:15:06.753947 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.746915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.6.53.125:35683)
0620 07:15:06.747107 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.747113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.747757 (+ 644us) server_negotiation.cc:407] Connection header received
0620 07:15:06.748010 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.748017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.748115 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.748222 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:06.748662 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.749827 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.751815 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.752469 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.752599 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.753056 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:06.753221 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:06.753321 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.753474 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.753579 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 07:15:06.767956 29410 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:06.766100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.6.53.124:40527)
0620 07:15:06.766299 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.766305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.766326 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:15:06.766553 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.766559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.766650 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.766724 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.767716 (+ 992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33848: BlockingRecv error: recv got EOF from 127.0.0.1:33848 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:15:06.782996 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.776172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.6.53.125:35683)
0620 07:15:06.776415 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.776421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.776918 (+ 497us) server_negotiation.cc:407] Connection header received
0620 07:15:06.777125 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.777132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.777220 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.777336 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:06.777790 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.778965 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.780903 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.781609 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.781763 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.782229 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:06.782409 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:06.782510 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.782646 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.782724 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260620 07:15:06.791246 29410 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:06.788972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.6.53.124:40527)
0620 07:15:06.789214 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.789220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.789783 (+ 563us) server_negotiation.cc:407] Connection header received
0620 07:15:06.789943 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.789948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.790053 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.790153 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.791026 (+ 873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33852: BlockingRecv error: recv got EOF from 127.0.0.1:33852 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 07:15:06.809221 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.802367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.6.53.125:35683)
0620 07:15:06.802530 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.802536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.803026 (+ 490us) server_negotiation.cc:407] Connection header received
0620 07:15:06.803184 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.803195 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.803300 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.803424 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:06.803872 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.805037 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.807064 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.807905 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.808008 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.808474 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:06.808586 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:06.808655 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.808828 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.808893 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 07:15:06.817719 29410 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:06.815908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.6.53.124:40527)
0620 07:15:06.816117 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.816123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.816416 (+ 293us) server_negotiation.cc:407] Connection header received
0620 07:15:06.816560 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.816565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.816658 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.816795 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.817488 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33862: BlockingRecv error: recv got EOF from 127.0.0.1:33862 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:15:06.843909 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.837766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50056 (local address 127.6.53.125:35683)
0620 07:15:06.838018 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.838025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.838371 (+ 346us) server_negotiation.cc:407] Connection header received
0620 07:15:06.838556 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.838568 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.838663 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.838783 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:06.839177 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.840393 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.841798 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.842530 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.842664 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.843137 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:06.843268 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:06.843366 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.843513 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.843617 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260620 07:15:06.852381 29410 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:06.850332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33868 (local address 127.6.53.124:40527)
0620 07:15:06.850539 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.850545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.850826 (+ 281us) server_negotiation.cc:407] Connection header received
0620 07:15:06.851017 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.851021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.851091 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.851173 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.852146 (+ 973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33868: BlockingRecv error: recv got EOF from 127.0.0.1:33868 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 07:15:06.894418 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.888385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50070 (local address 127.6.53.125:35683)
0620 07:15:06.888587 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.888594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.889015 (+ 421us) server_negotiation.cc:407] Connection header received
0620 07:15:06.889203 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.889209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.889319 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.889433 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:06.889839 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.891035 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.892362 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.893070 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.893200 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.893593 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:06.893732 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:06.893833 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.893961 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.894063 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 07:15:06.903164 29410 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:06.901087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33882 (local address 127.6.53.124:40527)
0620 07:15:06.901300 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.901306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.901636 (+ 330us) server_negotiation.cc:407] Connection header received
0620 07:15:06.901819 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.901823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.901910 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.901990 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.902889 (+ 899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33882: BlockingRecv error: recv got EOF from 127.0.0.1:33882 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 07:15:06.975519 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:06.970426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.6.53.125:35683)
0620 07:15:06.970648 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.970654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.970986 (+ 332us) server_negotiation.cc:407] Connection header received
0620 07:15:06.971134 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.971138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.971206 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.971344 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:06.971768 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.972522 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.973791 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:06.974198 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:06.974418 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:06.974826 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:06.974925 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:06.975032 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:06.975171 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 07:15:06.975247 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 07:15:06.984453 29410 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:06.982420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33894 (local address 127.6.53.124:40527)
0620 07:15:06.982668 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:06.982674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:06.982986 (+ 312us) server_negotiation.cc:407] Connection header received
0620 07:15:06.983169 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:06.983177 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:06.983278 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:06.983388 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:06.984232 (+ 844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33894: BlockingRecv error: recv got EOF from 127.0.0.1:33894 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 07:15:07.081013 29387 catalog_manager.cc:1259] Loaded cluster ID: a74cce3e758c448a9645731d6e8d9156
I20260620 07:15:07.081113 29387 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e: loading cluster ID for follower catalog manager: success
I20260620 07:15:07.083611 29387 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e: acquiring CA information for follower catalog manager: success
I20260620 07:15:07.085296 29387 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f549209beba94506bd36d9f91c02e52e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:15:07.121887 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:07.116627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50082 (local address 127.6.53.125:35683)
0620 07:15:07.116834 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:07.116838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:07.117265 (+ 427us) server_negotiation.cc:407] Connection header received
0620 07:15:07.117496 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:07.117502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:07.117602 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:07.117694 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:07.118132 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:07.118879 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:07.120187 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:07.120572 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:07.120660 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:07.121095 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:07.121209 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:07.121317 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:07.121514 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 07:15:07.121592 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 07:15:07.133678 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:07.127983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.6.53.124:40527)
0620 07:15:07.128192 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:07.128198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:07.128602 (+ 404us) server_negotiation.cc:407] Connection header received
0620 07:15:07.128750 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:07.128754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:07.128926 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:07.129022 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:07.129413 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:07.130126 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:07.131726 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:07.132100 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:07.132192 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:07.132650 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:07.133167 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:07.133273 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:07.133393 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 07:15:07.133464 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:15:07.149077 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:07.139438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.6.53.126:35971)
0620 07:15:07.141139 (+ 1701us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:07.141148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:07.141173 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:15:07.141371 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:07.141379 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:07.141639 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:07.141849 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:07.143005 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:07.144103 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:07.146319 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:07.147132 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:07.147283 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:07.147872 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:07.148349 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:07.148460 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:07.148555 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 07:15:07.148635 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1540,"thread_start_us":1424,"threads_started":1}
I20260620 07:15:07.149142 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:07.139589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33910 (local address 127.6.53.124:40527)
0620 07:15:07.139811 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:07.139819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:07.142691 (+ 2872us) server_negotiation.cc:407] Connection header received
0620 07:15:07.143034 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:07.143041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:07.143140 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:07.143204 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:07.143907 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:07.144939 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:07.147070 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:07.147702 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:07.147817 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:07.148347 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:07.148474 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:07.148556 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:07.148758 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 07:15:07.148836 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:15:07.149158 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:07.140559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.6.53.125:35683)
0620 07:15:07.140774 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:07.140782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:07.143295 (+ 2513us) server_negotiation.cc:407] Connection header received
0620 07:15:07.143465 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:07.143472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:07.143554 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:07.143646 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:07.144185 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:07.144929 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:07.147161 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:07.147829 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:07.147961 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:07.148426 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:07.148557 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:07.148637 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:07.148847 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 07:15:07.148931 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:15:07.154390 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 29155
I20260620 07:15:07.155333 29217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:15:07.266415 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 29155
I20260620 07:15:07.287245 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 29229
I20260620 07:15:07.288523 29291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:15:07.393908 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 29229
I20260620 07:15:07.413976 6357 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu pid 29303
I20260620 07:15:07.415118 29365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:15:07.519375 6357 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS8B4yM/build/asan/bin/kudu with pid 29303
2026-06-20T07:15:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4328 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8732 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (355473 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST